Slow on first page load

You did not mention what is the page you are first loading. Example a Moodle course, a Joomla article etc.

There could be many things causing the first page to load slowly and likely to be the cache is expiring.

Yes it does make sense sort of. It is still a little difficult to tell the exact problem as it could involve many things including software, connectivity, database, memory, hard disk, cpu, etc. Other parameters include shared vs VPS vs dedicate server hosting; system cache, module cache and browser cache.

You didn't mention the performance load without the calendar, how much data is on the calendar eg our you publishing events from hundreds of courses, whether you are logged in as a student viewing his/her data on the calendar or a guest; or what the page load time is (not page to load to your browser but the load time from the server which can be shown using many of the page load tools).

Sometimes you may find, especially on a shared server, that one extra plugin or module could throw the server over the edge in terms of performance when you are reaching the server's upper limit.

You must also realise that you are not just loading Joomla programmes and data but also Joomla programmes and Joomla and Moodle data which will increase the number of SQL queries. That said we have not experienced serious performance issues when loading the calendar.

Personally, I would disable the module and see the difference between first and subsequent loads. Check your module parameters for module cache, try loading a different Joomdle Module instead. Compare loading Joomla on its own, Moodle on its own and then Joomla with a Joomdle module.

Sorry I could not be of more assistance, at this point there are far too many parameters to narrow down the issue.