A powerful new suite of data products for evaluating solar irradiance and cloud cover across Australia are now available, developed through collaboration between CSIRO researchers and TERN data analysts. The 14 new products derived from geostationary Himawari-8/9 satellite data, gives users continent-wide insights on total solar radiation and cloud characteristics at ten-minute intervals. Together, they provide the data ‘building blocks’ for ecosystem research and environmental modelling and the reliable operation of Australia’s electricity grids.
How the light gets into a plant canopy
Downward Surface Solar Irradiance (DSSI) is the amount of incoming solar radiation at the Earth’s surface. It drives plant growth, evapotranspiration, and surface heating, which makes it an important input to many environmental models. Two key components of DSSI are the direct solar radiation and diffuse solar radiation, and they interact with vegetation on the Earth’s surface in different ways.
Direct solar radiation
Direct solar radiation accounts for most of the sunlight on bright, cloud-free days
- It is the sunlight coming directly from the sun;
- it has not been scattered by aerosols in the atmosphere on its journey to the Earth’s surface;
- In vegetation canopies, the upper leaves receive intense illumination, casting strong shadows on lower canopy layers;
- As a result, much of the canopy remains shaded, so fewer leaves are photosynthetically active.
Diffuse solar radiation
Diffuse solar radiation is sunlight scattered by dust, water vapour, aerosols and/or clouds
- On a fully cloudy day, all the sunlight received at the Earth’s surface can be diffuse solar radiation;
- Because diffuse light has been scattered at multiple angles, it’s distributed far more evenly than direct solar radiation;
- Diffuse solar radiation reduces the shadow cast by the upper canopy;
- Consequently, more leaves are photosynthetically active through the canopy layers (compared to a fully sunny day).
Image credits: Kyushu university/Kume lab
Why the balance between direct and diffuse light matters
Diffuse light enhances whole-canopy photosynthesis by increasing the proportion of solar radiation absorbed throughout a canopy. This boosts carbon uptake and influences ecosystem productivity. Understanding how much total, direct and diffuse solar radiation an ecosystem receives is therefore essential for modelling vegetation photosynthesis and gross primary productivity. Many plant growth and productivity models, including the widely used Light Use Efficiency (LUE) model, explicitly incorporate diffuse radiation estimates to calculate carbon biomass production.
A stable electricity grid
Knowing the amount of total incoming solar radiation as well as what proportion is direct or diffuse is also useful for maintaining stable electricity grids.
In 2025 Australia generated over 286 terawatt hours of electricity, nearly 20% of that was generated by solar energy. In fact, renewables (solar, wind, hydro) collectively accounted for nearly 40% of the nation’s electricity generation (Energy.gov.au). The continued adoption of solar energy across Australia and the world marks a meaningful move away from fossil fuels, but energy grid stability remains a significant obstacle to the renewable energy transition.
Electricity grid managers must carefully manage inflows and outflows of energy across the grid. The problem is that major renewable energy sources, such as solar and wind, can experience substantial fluctuations depending on the weather and/or time of day. As more renewables feed into an electricity distribution grid not designed to handle such fluctuations, the need to accurately predict surges and dips in energy is critical. In the case of solar energy, you need to be able to rapidly anticipate the amount of energy being produced by photovoltaic panels connected to the grid. To do that, you need to know how much and what type of solar irradiance is arriving at the solar panels.
Solar forecasting and the problem with clouds
Grid operators increasingly rely on solar forecasting to manage electrical loads. However, passing clouds cause small, rapid fluctuations in solar panel outputs that are difficult to predict at a local level, let alone across large, interconnected electricity grids spanning thousands of kilometres. The amount of solar energy produced depends not only on the presence or absence of clouds, but on their structure, pattern and opacity — and rapid changes in any of these cloud characteristics are especially hard to anticipate. As such, cloud cover detection and analysis are critical components of solar forecasting.
To detect and characterise cloud cover, analysts use a data processing technique called ‘cloud masking’ to identify which areas contain clouds and what type of clouds using the remote sensing imagery (Qin, Y. et al 2019). Reliable cloud masking is important for analysing remote sensing data for land and ocean applications.
Cloud masking algorithms have been developed based on the distinct spectral characteristics of clouds over the visible to thermal infrared spectral ranges, compared to the spectral characteristics of land and ocean surfaces. This has required parsing large volumes of data on cloud properties from a range of global remote sensing sources.
Nevertheless, cloud heterogeneity, movement, shadows, significant temporal variability in cloud and surface spectral data all introduce uncertainties that make accurate cloud masking difficult.
Top image: cloud cover over a solar panel array (image: Adobe iStock); Bottom image: clouds passing over an ecosystem (image: TERN)
A way forward
Recent advances in geostationary satellite technologies offer unparalleled opportunities to track cloud properties at a high temporal resolution and enhance the quality of cloud masking products. Geostationary satellites remain fixed over the same longitude (140.7° East) above the Earth’s equator (0.0° latitude) at about 35,800 kilometres, imaging the same location at regular intervals. The Himawari-8 and Himawari-9 satellites — launched in 2014 and 2016, respectively, by the Japanese Meteorological Agency (JMA) — gather remote sensing data over Asia and Oceania at 10-minute intervals, providing valuable imagery of the Australian continent and surrounding waters.
“The Himawari satellites have unprecedented temporal resolution as well as improved spatial and spectral resolutions compared to the previous generations of Japanese geostationary satellites and sensors,” says Tim McVicar, who is the TERN Himawari Project Leader with CSIRO Environment.
He explains that when it comes to cloud-spotting, the time-series of Himawari’s hyper-frequent data makes it easier to detect changes in clouds and identify the cause of those changes. For example, clouds and bright desert sand may look similar at the pixel level in a single image, but multiple images over time reveals movement, making it easier to distinguish clouds from land.
High frequency solar radiation and cloud products across the continent
For researchers at CSIRO, the Himawari-8/9 satellites presented a unique opportunity to develop a range of robust products to monitor sub-daily processes of Australia’s ecosystems and its overlying atmosphere. With this goal in mind, Tim McVicar, Tom Van Niel and Yi Qin developed a TERN Himawari project and collaborated with Dejun (Jack) Cai and Robert Brigart from CSIRO, along with the Australian Bureau of Meteorology (BoM) and University staff to produce a series of products adapted for Himawari-8/9 data to monitor Australia’s ecosystems.
The work began with the development of a cloud masking algorithm that automatically detects daytime clouds in satellite imagery. It was originally tested on imagery from the European Space Agency’s Advanced Along-Track Scanning Radiometer (AATSR). Following encouraging results, the team optimised the algorithm for Himawari data and validating it against six years of spaceborne LiDAR data. In doing so, they confirmed the algorithm could detect cloud cover with an overall accuracy of 98% (Qin, Y. et al 2019). Using BoM’s high-quality network of stations measuring solar radiation, they developed new algorithms optimised for Himawari data to model sunlight characteristics at an unprecedented 10-minute time-step for all of Australia.
Side-by-side animations of the total incoming solar radiation over Australia at 10-minute time intervals. On left: time-lapse spatial dynamics; On right: 10-minute time series chart of total radiation at Sydney and Perth over the same period. For the left image for day 1 from 01:00 UTC (10 second mark) to 05:00 UTC (17 second mark) the low solar radiation associated with north-west cloud band straddling the SA-WA border is clearly seen as are small convective clouds appearing over much of the NT, western Qld and northwest NSW. For the right image there are three main things to note: (i) as expected, the sun rises earlier in the east (Sydney) and sets later in the west (Perth); (ii) scattered clouds result in highly dynamic sunlight for the morning of day 1 for both Sydney and Perth, and the day 2 afternoon at Sydney; and (iii) day 2 for Perth is cloud-free and so the solar radiation essentially follows a sine curve mainly governed by atmospheric path length (source: Dejun (Jack) Cai).
In total, the team used these new algorithms to develop 14 Himawari products that evaluate solar radiation and cloud cover across the Australian region. These new products deliver a powerful high-frequency monitoring tool, that will be useful to a wide range of users.
The cloud cover masks can be used to ensure the next suite of Himawari products — including biophysical variables like Leaf Area Index (LAI), fraction vegetation cover and albedo — can be cloud masked to avoid artefacts being introduced in ecosystem analysis / biophysical modelling. Additionally, atmospheric scientists can use the suite of Himawari cloud products to quantify if cloud presence / absence, cloud type and cloud properties are changing in our changing climate.
The fine temporal resolution is will also provide a significant advantage to power grid operators, says Tim.
“Knowing solar loading on a 10-minute time-step enables much better estimates of the amount of energy that can be produced from photovoltaic panels,” he explains. “This will improve the ability to monitor and predict solar energy fluctuations. It’s going to be very useful.”
14 new Himawari products
Daytime Cloud Type for Himawari-8/9 (HIM_CLD_Day_Type)
Daytime Cloud Mask for Himawari-8/9 (HIM_CLD_Day_Mask)
Daytime Cloud Optical Depth for Himawari-8/9 (HIM_CLD_Day_OD)
Directly Transmitted Downward Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_Direct)
Diffusely Transmitted Downward Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_Diffuse)
Total Downward PAR Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_PAR_Total)
Total Net Downward Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_Netdown)
Total Downward Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_Total)
Directly Transmitted Downward PAR Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_PAR_Direct)
Diffusely Transmitted Downward PAR Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_PAR_Diffuse)
Total Net Downward PAR Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_PAR_Netdown)
Daily Exposure of Total Downward Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_DE_Total)
Daily Exposure of Directly Transmitted Downward Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_DE_Direct)
Daily Exposure of Diffusely Transmitted Downward Surface Solar Irradiance for Himawari-8/9 (HIM_SSI_DE_Diffuse)
Streamlining Data Product Access and Use from TERN
The CSIRO team worked closely with TERN data systems specialists to make the Himawari products available on TERN’s Data Discovery Portal and ensuring file formats are optimal for user access.
There are three main ways to access these Himawari cloud and solar radiation data products from the TERN Data Portal:
- Downloading / reading the NetCDF files directly via HTTP
- Reading the NetCDF files via the TERN THREDDS server
- Reading the data using Python kerchunk*
The first method is the most straight-forward: the 10-minutely data products are packaged into daily NetCDF files which can be downloaded individually or read over the Internet using GIS software/libraries.
The second involves using TERN’s THREDDS data server, which provides both standard read access and server-side sub-setting of the individual daily NetCDF files.
The third, and most powerful if using Python, is to use the kerchunk parquet index file which can be read over the Internet and which provides an overview of the entire product as one giant dataset. Each entire product is indexed as numerous small chunks of data – with only those chunks required by the user being downloaded in parallel, providing fast and efficient access to the data.
It’s important to note that this work was also made possible thanks to national and international collaborations across the research ecosystem. This includes an international scientific data agreement between JMA and BoM where Himawari data are provided to Australia via a dedicated internet link from Japan to the BoM for operational use. BoM make these observations available to research agencies like CSIRO, Geoscience Australia (GA) and Australia’s university sector via the NCRIS-funded National Computational Infrastructure (NCI). The NCI high-performance computing environment enables researchers to process Himawari data volumes in near real-time, including the processing of all 14 new Himawari products.
Feature image: True colour composite satellite image of Australia. Image taken on 17 June 2026 by the Himawari-9 geostationary weather satellite operated by the Japan Meteorological Agency (JMA) accessed via the Australian BoM Satellite Viewer. White areas are cloud, the surrounding oceans are blue, and land areas are brown / red to green depending on the amount of vegetation cover. Brown / red areas have low amounts of vegetation cover whereas green areas have high amounts of vegetation cover.