13:08:28 From Cristina Ruiz Villena (NCEO - U. Leicester) to Everyone: xarray is wonderful, it's one of my favourite Python packages ever 13:08:37 From Hydro-JULES Education to Everyone: Reacted to "xarray is wonderful,..." with πŸ‘ 13:08:38 From Kit Macleod to Everyone: Reacted to "xarray is wonderful,..." with πŸ‘ 13:08:41 From Amulya Chevuturi to Everyone: Reacted to "xarray is wonderful,..." with πŸ‘ 13:08:43 From NG YU EN to Everyone: Reacted to "xarray is wonderful,..." with ❀️ 13:08:52 From Fatima D to Everyone: Reacted to xarray is wonderful,... with "πŸ‘" 13:08:54 From Fatima D to Everyone: Removed a πŸ‘ reaction from "xarray is wonderful,..." 13:09:02 From Amulya Chevuturi to Everyone: As in the last workshop, please feel free to ask any questions. 13:09:11 From Fatima D to Everyone: Reacted to xarray is wonderful,... with "πŸ‘" 13:09:22 From Cristina Ruiz Villena (NCEO - U. Leicester) to Everyone: Reacted to "As in the last works..." with πŸ‘ 13:09:29 From Cuadros Adriazola, Jose to Everyone: I had some issues when using xarray. I'll try to put an example. Say I have a 3d (x,y,time) xarray and I have created another 2d object (which could be a mask or something like that). Whenever I tried to do numpy calculations between the 3d and 2d arrays, I got problems. I think there is something I am missing. How does xarray handle arrays with different dimensions? In the past, I just replicated my 2d to convert it into a 3d with the same dimensions of the 3d. 13:10:14 From Amulya Chevuturi to Everyone: Replying to "I had some issues wh..." I will reply to you, just give me a minute. 13:10:30 From Cuadros Adriazola, Jose to Everyone: Reacted to "I will reply to you,..." with ❀️ 13:12:14 From Matthew Fry to Everyone: https://pangeo.io/ 13:13:31 From Kit Macleod to Everyone: Reacted to "I will reply to you,..." with ❀️ 13:13:40 From Kit Macleod to Everyone: Reacted to "https://pangeo.io/" with πŸ‘ 13:13:51 From Matthew Fry to Everyone: Good background here for later reading, relating to what Amulya is describing: https://pangeo.io/about.html 13:19:45 From NG YU EN to Everyone: Reacted to "Good background here..." with πŸ‘ 13:19:48 From NG YU EN to Everyone: Reacted to "https://pangeo.io/" with πŸ‘ 13:20:58 From Xiancheng Lu to Everyone: How large a metadata can be read and worked in the Jupyter notebook? 13:21:01 From Kit Macleod to Everyone: Some of the Pangeo videos compare speeds (119) Pangeo - YouTube 13:21:32 From Amulya Chevuturi to Everyone: Object store GitHub repository: https://github.com/NERC-CEH/object_store_tutorial/ 13:26:31 From Matthew Fry to Everyone: As Amulya says this technology is relatively new but it's likely to be (and already is) an important way to access this type of data in future, so we thought it was useful to give an intro. 13:26:47 From Cristina Ruiz Villena (NCEO - U. Leicester) to Everyone: Reacted to "As Amulya says this ..." with ❀️ 13:27:08 From Amulya Chevuturi to Everyone: Replying to "I had some issues wh..." You are absolutely right, that if you need to multiply 2D data with 3D data in numpy you have to add an axis by replication or np.newaxis() function. However, in xarray adding an axis can be a little more involved or complicated. I usually go about this old school by looping through extra dimension. Or I do it by converting the xarray array to numpy array and then multiplying it. Does that make sense? 13:27:16 From Amulya Chevuturi to Everyone: Reacted to "As Amulya says this ..." with ❀️ 13:30:04 From Cuadros Adriazola, Jose to Everyone: Replying to "I had some issues wh..." Yes, thanks! That are some options that I used before, I was just wondering if there were some additional features that could make it easier in xarray (I think there is some broadcasting rules but I am not completely familiar with this). 13:31:20 From Amulya Chevuturi to Everyone: Replying to "I had some issues wh..." To be fair, I am also not. But let me explore a bit and get back to you. thanks. 13:31:29 From Cuadros Adriazola, Jose to Everyone: Reacted to "To be fair, I am als..." with πŸ‘ 13:32:14 From Amulya Chevuturi to Everyone: Replying to "How large a metadata..." Reading metadata is never the problem, as that is usually very small comparatively. The problem is always the reading the data itself. 13:32:29 From NG YU EN to Everyone: Reacted to "As Amulya says this ..." with ❀️ 13:36:04 From Amulya Chevuturi to Everyone: Lazy loading is very important when combined with DASK and chunking. This makes the data analysis via network very efficient. 13:39:28 From Amulya Chevuturi to Everyone: Replying to "How large a metadata..." To answer your exact question, it depends memory on the computer on which the Jupyter Notebook is hosted on. 13:39:36 From Cuadros Adriazola, Jose to Hydro-JULES Education(direct message): Hi Alanna. I was just curious. Was there a tentative month/year for the intermediate JULES course? 13:40:00 From Hydro-JULES Education to Cuadros Adriazola, Jose(direct message): Hi Jose, yes they are thinking March/April 13:40:18 From Cuadros Adriazola, Jose to Hydro-JULES Education(direct message): Thanks! 13:40:26 From Hydro-JULES Education to Cuadros Adriazola, Jose(direct message): i have you on my list so we will let you know as soon as more details are available 13:40:59 From Cuadros Adriazola, Jose to Hydro-JULES Education(direct message): Reacted to "i have you on my lis..." with πŸ‘ 13:42:07 From Xiancheng Lu to Everyone: Reacted to "To answer your exact..." with πŸ‘ 13:42:10 From Matthew Fry to Everyone: https://cosmos.ceh.ac.uk/ website and map of sites where you can visualise the data (on the time series tab): https://cosmos.ceh.ac.uk/data 13:42:25 From Xiancheng Lu to Everyone: Reacted to "Lazy loading is very..." with πŸ‘ 13:42:44 From Xiancheng Lu to Everyone: Reacted to "Reading metadata is ..." with πŸ‘ 13:42:59 From Hydro-JULES Education to Everyone: As Matt mentioned, we will be hearing more on COSMOS and other large monitoring networks in the lighting talks tomorrow morning 13:43:02 From Hydro-JULES Education to Everyone: Reacted to "https://cosmos.ceh.a..." with πŸ‘ 13:43:06 From Matthew Fry to Everyone: Reacted to "As Matt mentioned, w..." with πŸ‘ 13:43:10 From Xiancheng Lu to Everyone: Reacted to "As Matt mentioned, w..." with πŸ‘ 13:43:12 From Xiancheng Lu to Everyone: Reacted to "https://cosmos.ceh.a..." with πŸ‘ 13:43:18 From Matthew Fry to Everyone: Replying to "https://cosmos.ceh.a..." API docs: https://cosmos-api.ceh.ac.uk/docs 13:44:37 From NG YU EN to Everyone: Reacted to "As Matt mentioned, w..." with πŸ‘ 13:45:05 From Amulya Chevuturi to Everyone: Reacted to "As Matt mentioned, w..." with πŸ‘ 13:45:14 From Amulya Chevuturi to Everyone: Reacted to "API docs: https://co..." with πŸ‘ 13:46:03 From Cuadros Adriazola, Jose to Everyone: Is JASMIN object store accesible to everyone or should we be logged into a JASMIN server? 13:46:05 From Matthew Fry to Everyone: Replying to "https://cosmos.ceh.a..." Also available from the International Soil Moisture Network along with other soil moisture data: https://ismn.earth/en/ 13:48:50 From Matthew Fry to Everyone: Replying to "Is JASMIN object sto..." This dataset is (I think) accessible to anyone over the internet. Specific permissions are set for different datasets / buckets. By default it's internal to Jasmin but one of its benefits is that it can be shared over the web. 13:49:13 From Amulya Chevuturi to Everyone: Replying to "Is JASMIN object sto..." Unfortunately, JASMIN object store is only available for people with JASMIN accounts. This particular dataset is open access, so you can access this even when your JASMIN training accounts close. 13:49:32 From Amulya Chevuturi to Everyone: Reacted to "This dataset is (I t..." with πŸ‘ 13:49:37 From Amulya Chevuturi to Everyone: Reacted to "Also available from ..." with πŸ‘ 13:50:46 From Matthew Fry to Everyone: Replying to "Is JASMIN object sto..." Again, I suspect that as use of the object store grows there will be more (properly catalogued) datasets on there that are publicly available. It's a good way to share these large datasets particularly for people who just want to extract a specific subset. 13:50:57 From Amulya Chevuturi to Everyone: Reacted to "Again, I suspect tha..." with πŸ‘ 13:51:25 From Matthew Fry to Everyone: Replying to "Is JASMIN object sto..." In the FDRI digital work we are hoping to take forward the way we use this (e.g. for UK rainfall datasets) 13:51:39 From Amulya Chevuturi to Everyone: Reacted to "In the FDRI digital ..." with πŸ‘ 13:51:39 From Cuadros Adriazola, Jose to Everyone: Reacted to "Unfortunately, JASMI..." with πŸ‘ 13:51:42 From Cuadros Adriazola, Jose to Everyone: Reacted to "This dataset is (I t..." with πŸ‘ 13:51:53 From Cuadros Adriazola, Jose to Everyone: Reacted to "Again, I suspect tha..." with πŸ‘ 13:52:09 From Gemma Coxon to Everyone: Reacted to "In the FDRI digital ..." with ❀️ 13:52:44 From Amulya Chevuturi to Everyone: Replying to "Is JASMIN object sto..." If anyone is interested about this more beyond JASMIN, please see https://pangeo.io/index.html the Pangeo Community. 13:54:08 From Kit Macleod to Everyone: dynamical-org 13:54:20 From Kit Macleod to Everyone: dynamical.org - home 13:56:55 From Kit Macleod to Everyone: STAC -> Xarray Loading Best Practices - Data - Pangeo 13:57:31 From Kit Macleod to Everyone: opendatacube/odc-geo: GeoBox and geometry utilities extracted from datacube-core 13:58:19 From Kit Macleod to Everyone: Welcome to rioxarray’s documentation! β€” rioxarray 0.16.0 documentation 13:59:03 From Neven Fuckar (Oxford University) to Everyone: ncdump command in Linux/Unix can be of help 14:00:02 From Hydro-JULES Education to Everyone: thanks for your question Stevan and great explanation, Amulya! 14:00:05 From Hydro-JULES Education to Everyone: Reacted to "ncdump command in Li..." with πŸ‘ 14:00:32 From Amulya Chevuturi to Everyone: More resources from the Pangeo Community: https://docs.earthmover.io/ https://pangeo.io/index.html https://www.coiled.io/ 14:01:56 From Matthew Fry to Everyone: https://www.cogeo.org/ 14:02:57 From Kit Macleod to Everyone: Matt shared this example where they are exploring intake and STAC catalogue options Data Concept β€” ORCESTRA (orcestra-campaign.org) 14:03:20 From Matthew Fry to Everyone: Replying to "https://www.cogeo.or..." Cloud optimised Geotiff 14:04:22 From Cuadros Adriazola, Jose to Everyone: Didn't know about that one, thanks Amulya! 14:04:28 From Matthew Fry to Everyone: Reacted to "Matt shared this exa..." with πŸ‘ 14:04:32 From Matthew Fry to Everyone: Object store GitHub repository: https://github.com/NERC-CEH/object_store_tutorial/ 14:05:21 From Kit Macleod to Everyone: https://cf-xarray.readthedocs.io/en/latest/ 14:05:43 From Matthew Fry to Everyone: This website can be used to chec kif a netCDF file has been created according to CF conventions, both the structure and the mandatory metadata: https://cfchecker.ncas.ac.uk/ 14:05:49 From Amulya Chevuturi to Everyone: Reacted to "https://cf-xarray.re..." with πŸ‘ 14:05:56 From Amulya Chevuturi to Everyone: Reacted to "This website can be ..." with πŸ‘ 14:05:59 From Amulya Chevuturi to Everyone: Reacted to "Didn't know about th..." with ❀️ 14:06:03 From Matthew Fry to Everyone: Replying to "This website can be ..." But I usually start by basing it on an existing one 14:06:41 From Hydro-JULES Education to Everyone: I will just say, I'm collating all the useful links and resources in the chat so don't worry about copy-ing and pasting everything. These will be posted on the resource page 14:07:14 From Kit Macleod to Everyone: Reacted to "I will just say, I'm..." with πŸ‘ 14:07:23 From NG YU EN to Everyone: Reacted to "I will just say, I'm..." with ❀️ 14:07:23 From Daira AnahΓ­ Rosales to Everyone: Reacted to "I will just say, I'm..." with ❀️ 14:08:26 From Kit Macleod to Everyone: JASMIN Help Site - Jaspy Software Environments (Python 3, R and other tools) 14:09:02 From Kit Macleod to Everyone: ceda-jaspy-envs/environments/py3.11/mf3-23.11.0-0/jaspy3.11-mf3-23.11.0-0-r20240508/final-spec.yml at main Β· cedadev/ceda-jaspy-envs 14:09:26 From Daira AnahΓ­ Rosales to Everyone: Thank you very much 14:09:28 From Neven Fuckar (Oxford University) to Everyone: Thank you! 14:09:29 From Gemma Coxon to Everyone: Thank you all - excellent session! 14:09:32 From Asid Ur Rehman to Everyone: Thanks 14:09:35 From Flavia Ribeiro-EACH to Everyone: Thank you! 14:09:35 From Cristina Ruiz Villena (NCEO - U. Leicester) to Everyone: thank you! 14:09:49 From Hydro-JULES Education to Everyone: https://ukceh-ac-uk.zoom.us/j/91808553577?pwd=0JqyvPnFaBfIsVf98X8fGGGpds85jY.1 14:09:50 From Kallista Angeloff to Everyone: Thank you! 14:09:56 From NG YU EN to Everyone: Thank you!!