Contents

EcoFOCI evaluation/application of Glider Tools for post processing

Summary

Resources

Oculus Glider Data Applicability

There are pre-existing QC procedures built into both, SeaGlider data (what GliderTools works with) and Oculus data as developed by UW.

There are Engineering parameters and science parameters (including platform instrumentation and additional science instrumentation)

Oculus Glider Data Visualization (PMEL local or FOCI NRT)

  • Kongsberg pilot webpage (immediate information but only for pilots and PI’s)

  • internal plotly dashboard (NRT / NRT QC) - of binned profile data or timeseries data… quick to plot and look at

  • internal plotly dashboard (Delayed Mode QC), see gliderTools + IOOS QARTOD + PMEL QC

  • GliderScope??? https://imos.org.au/gliderscope (gui third party-matlab based runtime-needs admin to install on Mac but also is available for Win)

Other Services and gliders

  • https://gliders.ioos.us/map/

Oculus Glider Distribition

Via Internal ERDDAP:

  • NRT (Pre-existing QC from timeseries files or profile files made by the basestation)

    • profile files are binned by depth intervals (5m) and may be better hosted as a “gridded” dataset via erddap than tabular

    • timeseries is one long datastream

  • Processed and Cleaned

    • initial qc with improved dials?

    • GliderTools

    • IOOS QARTOD QC

  • Modified Science Routines

    • BeringSea interface tools

Via IOOS GliderDAC

Examples from BS long mission (2017)

Local erddap has two datastreams available for simplicity… the timeseries merged profiles data and the upanddown merged profiles data (from the base station).

Start with gridded (5m) basestation data (NRT level quicklooks with some on base station QC)

from erddapy import ERDDAP
import pandas as pd

server_url = 'http://akutan.pmel.noaa.gov:8080/erddap'

e = ERDDAP(server=server_url)
# timeseries or upanddown (profiles)
df = pd.read_csv(e.get_search_url(response='csv', search_for='datasets_sg401_2017_'))
df
griddap Subset tabledap Make A Graph wms files Title Summary FGDC ISO 19115 Info Background Info RSS Email Institution Dataset ID
0 NaN http://akutan.pmel.noaa.gov:8080/erddap/tabled... http://akutan.pmel.noaa.gov:8080/erddap/tabled... http://akutan.pmel.noaa.gov:8080/erddap/tabled... NaN http://akutan.pmel.noaa.gov:8080/erddap/files/... Physical data collected from Seaglider SG401 d... PMEL SG401 Bering Sea Test Apr 26, 2017. Up an... http://akutan.pmel.noaa.gov:8080/erddap/metada... http://akutan.pmel.noaa.gov:8080/erddap/metada... http://akutan.pmel.noaa.gov:8080/erddap/info/d... https://data.nodc.noaa.gov/accession/0092291 http://akutan.pmel.noaa.gov:8080/erddap/rss/da... http://akutan.pmel.noaa.gov:8080/erddap/subscr... NOAA/PMEL datasets_sg401_2017_upanddown
1 NaN http://akutan.pmel.noaa.gov:8080/erddap/tabled... http://akutan.pmel.noaa.gov:8080/erddap/tabled... http://akutan.pmel.noaa.gov:8080/erddap/tabled... NaN http://akutan.pmel.noaa.gov:8080/erddap/files/... Physical data collected from Seaglider SG401 d... PMEL SG401 Bering Sea Test Apr 26, 2017. Times... http://akutan.pmel.noaa.gov:8080/erddap/metada... http://akutan.pmel.noaa.gov:8080/erddap/metada... http://akutan.pmel.noaa.gov:8080/erddap/info/d... https://data.nodc.noaa.gov/accession/0092291 http://akutan.pmel.noaa.gov:8080/erddap/rss/da... http://akutan.pmel.noaa.gov:8080/erddap/subscr... NOAA/PMEL - EcoFOCI datasets_sg401_2017_timeseries
2 http://akutan.pmel.noaa.gov:8080/erddap/gridda... NaN NaN http://akutan.pmel.noaa.gov:8080/erddap/gridda... NaN http://akutan.pmel.noaa.gov:8080/erddap/files/... Physical, chemical, and biological data collec... SG401 Bering Sea. Applied Physics Lab\n ... NaN NaN http://akutan.pmel.noaa.gov:8080/erddap/info/d... https://data.nodc.noaa.gov/accession/0092291 http://akutan.pmel.noaa.gov:8080/erddap/rss/da... http://akutan.pmel.noaa.gov:8080/erddap/subscr... Applied Physics Lab University of Washington S... datasets_sg401_2017_upanddown_gridded 
#thin convenience wrapper for loading FOCI erddap glider data
import GliderDataLoader

sg401=GliderDataLoader.erddap_glider('datasets_sg401_2017_upanddown')
sg401.list_data(verbose=True)

0            datasets_sg401_2017_upanddown
1    datasets_sg401_2017_upanddown_gridded
Name: Dataset ID, dtype: object

variables = {'datasets_sg401_2017_upanddown':['profile_id',
                     'latitude', 
                     'depth', 
                     'longitude', 
                     'salinity',
                     'density', 
                     'time', 
                     'temperature']}
sg401.constrain_data(variables=variables,start_date='2017-01-01T00:00:00Z')
dfs = sg401.load_data(year='2017')

datasets_sg401_2017_upanddown
datasets_sg401_2017_upanddown_gridded
Failed to generate url datasets_sg401_2017_upanddown_gridded

sg401.plot_timeseries(dfs['datasets_sg401_2017_upanddown'].sort_index(), 'temperature (degree_C)','depth (m)','temperature (DegC)')

(<Figure size 1224x144 with 2 Axes>, <AxesSubplot:ylabel='Depth (m)'>)
../../../_images/EcoFOCI_GliderTools_Readme_9_1.png

Use Gridded ERRDAP Dataset if Available for binned data

#all
import cmocean
e.protocol = 'griddap'
e.dataset_id='datasets_sg401_2017_upanddown_gridded'
e.variables = ['temperature','ctd_time']
e.response = 'ncCF'
ds = e.to_xarray(decode_times=True)

ds.temperature.T.plot(yincrease=False,cmap=cmocean.cm.thermal,vmin=-2,vmax=12,figsize=(18,2))

<matplotlib.collections.QuadMesh at 0x7fb4dbbc5fa0>
../../../_images/EcoFOCI_GliderTools_Readme_12_1.png

Now using larger archived timeseries data ((NRT level quicklooks with some on base station QC)

sg401=GliderDataLoader.erddap_glider('datasets_sg401_2017_timeseries')
sg401.list_data(verbose=True)

0    datasets_sg401_2017_timeseries
Name: Dataset ID, dtype: object

variables = {'datasets_sg401_2017_timeseries':['profile_id',
                     'latitude', 
                     'depth', 
                     'longitude', 
                     'salinity',
                     'density', 
                     'time', 
                     'temperature']}
sg401.constrain_data(variables=variables,start_date='2017-01-01T00:00:00Z')
dfs = sg401.load_data(year='2017')

datasets_sg401_2017_timeseries

sg401.plot_timeseries(dfs['datasets_sg401_2017_timeseries'].sort_index(), 'temperature (degree_C)','depth (m)','temperature (DegC)')

(<Figure size 1224x144 with 2 Axes>, <AxesSubplot:ylabel='Depth (m)'>)
../../../_images/EcoFOCI_GliderTools_Readme_16_1.png

Loading individual dive datasets and running GliderTools

This isn’t truly Delayed mode data (just delayed mode processing)

%load_ext autoreload
# pylab for more MATLAB like environment and inline displays plots below cells 
%pylab inline

# if gsw Warning shows, manually install gsw if possible - will still work without
import glidertools as gt
from cmocean import cm as cmo  # we use this for colormaps

Populating the interactive namespace from numpy and matplotlib

Loading data

Working with Seaglider base station files

GliderTools supports loading Seaglider files, including scicon data (different sampling frequencies).
There is a function that makes it easier to find variable names that you’d like to load: gt.load.seaglider_show_variables

This function is demonstrated in the cell below. The function accepts a list of file names and can also receive a string with a wildcard placeholder (*) and basic regular expressions are also supported. In the example below we use a simple asterisk placeholder for all the files.

Note that the function chooses only one file from the passed list or glob string - this file name will be shown. The returned table shows the variable name, dimensions, units and brief comment if it is available.

#skipp first 1000 dives
filenames = 'test_data/sg401_2017/p401[123]*.nc'

gt.load.seaglider_show_variables(filenames)

information is based on file: test_data/sg401_2017/p4012283.nc
  dims units comment
name      
gc_data_pts gc_event 1
gc_depth gc_event meters
gc_end_secs gc_event seconds since 1970-1-1 00:00:00 End of GC time in GMT epoch format
gc_gcphase gc_event
gc_humidity gc_event percent Internal relative humidity
gc_int_press gc_event psia Internal pressure
gc_ob_vertv gc_event cm/s
gc_pitch_ad gc_event 1
gc_pitch_ad_start gc_event 1
gc_pitch_ctl gc_event cm
gc_pitch_errors gc_event 1
gc_pitch_i gc_event A
gc_pitch_retries gc_event 1
gc_pitch_secs gc_event seconds Elapsed seconds since start of this pitch change
gc_pitch_volts gc_event V
gc_roll_ad gc_event 1
gc_roll_ad_start gc_event 1
gc_roll_errors gc_event 1
gc_roll_i gc_event A
gc_roll_retries gc_event 1
gc_roll_secs gc_event seconds Elapsed seconds since start of this roll change
gc_roll_volts gc_event V
gc_st_secs gc_event seconds since 1970-1-1 00:00:00 Start of GC time in GMT epoch format
gc_vbd_ad gc_event 1
gc_vbd_ctl gc_event cm
gc_vbd_errors gc_event 1
gc_vbd_i gc_event A
gc_vbd_pot1_ad gc_event 1
gc_vbd_pot1_ad_start gc_event 1
gc_vbd_pot2_ad gc_event 1
gc_vbd_pot2_ad_start gc_event 1
gc_vbd_retries gc_event 1
gc_vbd_secs gc_event seconds Elapsed seconds since start of this VBD change
gc_vbd_volts gc_event V
log_gps_lat gps_info degrees_north GPS latitudes
log_gps_lon gps_info degrees_east GPS longitudes
log_gps_time gps_info seconds since 1970-1-1 00:00:00 GPS times in GMT epoch format
eng_mag_x magnetometer_data_point magnetometer x
eng_mag_y magnetometer_data_point magnetometer y
eng_mag_z magnetometer_data_point magnetometer z
eng_satd_PARuV sat1000_data_point
eng_satd_Temp sat1000_data_point
eng_satd_Timer sat1000_data_point
eng_satd_pitch sat1000_data_point
eng_satd_roll sat1000_data_point
eng_satu_PARuV sat1001_data_point
eng_satu_Temp sat1001_data_point
eng_satu_Timer sat1001_data_point
eng_satu_pitch sat1001_data_point
eng_satu_roll sat1001_data_point
aanderaa4330_dissolved_oxygen sg_data_point micromoles/kg Dissolved oxygen concentration, calculated from optode tcphase corrected for salininty and depth
aanderaa4330_dissolved_oxygen_qc sg_data_point Whether to trust each optode dissolved oxygen value
aanderaa4330_instrument_dissolved_oxygen sg_data_point micromoles/kg Dissolved oxygen concentration reported from optode corrected for salinity
aanderaa4330_results_time sg_data_point seconds since 1970-1-1 00:00:00 time for Aanderaa 4330 in GMT epoch format
buoyancy sg_data_point g Buoyancy of vehicle, corrected for compression effects
conductivity sg_data_point S/m Conductivity corrected for anomalies
conductivity_qc sg_data_point Whether to trust each corrected conductivity value
conductivity_raw sg_data_point S/m Uncorrected conductivity
conductivity_raw_qc sg_data_point Whether to trust each raw conductivity value
ctd_depth sg_data_point meters CTD thermistor depth corrected for average latitude
ctd_pressure sg_data_point dbar Pressure at CTD thermistor
ctd_time sg_data_point seconds since 1970-1-1 00:00:00 Time of CTD sample in GMT epoch format
density sg_data_point g/m^3 Sea water potential density
density_insitu sg_data_point g/m^3 Sea water in-situ density based on pressure
depth sg_data_point meters Depth below the surface, corrected for average latitude
dissolved_oxygen_sat sg_data_point micromoles/kg Calculated saturation value for oxygen given measured presure and corrected temperature, and salinity
east_displacement sg_data_point meters Eastward displacement from hdm
east_displacement_gsm sg_data_point meters Eastward displacement from gsm
eng_GC_phase sg_data_point
eng_aa4330_AirSat sg_data_point As reported by the instrument
eng_aa4330_CalPhase sg_data_point As reported by the instrument
eng_aa4330_O2 sg_data_point micromoles/L Dissolved oxygen as reported by the instument, based on on-board calibration data, assuming optode temperature but without depth or salinity correction
eng_aa4330_TCPhase sg_data_point As reported by the instrument
eng_aa4330_Temp sg_data_point degrees_Celsius As reported by the instrument
eng_depth sg_data_point cm Measured vertical distance below the surface
eng_elaps_t sg_data_point seconds Elapsed seconds since start of dive
eng_elaps_t_0000 sg_data_point seconds Elapsed seconds since start of mission
eng_head sg_data_point degrees Vehicle heading (magnetic)
eng_pitchAng sg_data_point degrees Vehicle pitch
eng_pitchCtl sg_data_point
eng_rec sg_data_point
eng_rollAng sg_data_point degrees Vehicle roll
eng_rollCtl sg_data_point
eng_sbect_condFreq sg_data_point As reported by the instrument
eng_sbect_tempFreq sg_data_point As reported by the instrument
eng_vbdCC sg_data_point
eng_wlbb2fl_sig470nm sg_data_point blue scattering as reported by instrument
eng_wlbb2fl_sig695nm sg_data_point chlorophyll fluorescence as reported by instrument
eng_wlbb2fl_sig700nm sg_data_point red scattering as reported by instrument
eng_wlbb2fl_temp sg_data_point temperature as reported by instrument
glide_angle sg_data_point cm/s Glide angle based on hdm
glide_angle_gsm sg_data_point cm/s Glide angle based on gsm
horz_speed sg_data_point cm/s Vehicle horizontal speed based on hdm
horz_speed_gsm sg_data_point cm/s Vehicle horizontal speed based on gsm
latitude sg_data_point degrees_north Latitude of the sample based on hdm DAC
latitude_gsm sg_data_point degrees_north Latitude of the sample based on gsm DAC
longitude sg_data_point degrees_east Longitude of the sample based on hdm DAC
longitude_gsm sg_data_point degrees_east Longitude of the sample based on gsm DAC
north_displacement sg_data_point meters Northward displacement from hdm
north_displacement_gsm sg_data_point meters Northward displacement from gsm
pressure sg_data_point dbar Uncorrected sea-water pressure at pressure sensor
salinity sg_data_point 1e-3 Salinity corrected for thermal-inertia effects (PSU)
salinity_qc sg_data_point Whether to trust each corrected salinity value
salinity_raw sg_data_point 1e-3 Uncorrected salinity derived from temperature_raw and conductivity_raw (PSU)
salinity_raw_qc sg_data_point Whether to trust each raw salinity value
sigma_t sg_data_point g/m^3 Sigma based on density
sigma_theta sg_data_point g/m^3
sound_velocity sg_data_point m/s Sound velocity
speed sg_data_point cm/s Vehicle speed based on hdm
speed_gsm sg_data_point cm/s Vehicle speed based on gsm
speed_qc sg_data_point Whether to trust each hdm speed value
temperature sg_data_point degrees_Celsius Termperature (in situ) corrected for thermistor first-order lag
temperature_qc sg_data_point Whether to trust each corrected temperature value
temperature_raw sg_data_point degrees_Celsius Uncorrected temperature (in situ)
temperature_raw_qc sg_data_point Whether to trust each raw temperature value
theta sg_data_point degrees_Celsius Potential temperature based on corrected salinity
time sg_data_point seconds since 1970-1-1 00:00:00 Time of the sample in GMT epoch format
vert_speed sg_data_point cm/s Vehicle vertical speed based on hdm
vert_speed_gsm sg_data_point cm/s Vehicle vertical speed based on gsm
wlbb2fl_results_time sg_data_point seconds since 1970-1-1 00:00:00 wlbb2fl result time in GMT epoch format
wlbb2fl_sig470nm_adjusted sg_data_point meter^-1 steradian^-1 total volume blue scattering coefficient using manufacturer-supplied dark counts and scaling factor
wlbb2fl_sig695nm_adjusted sg_data_point micrograms/liter chlorophyll-a concentration using manufacturer-supplied dark counts and scaling factor based on phytoplankton monoculture
wlbb2fl_sig700nm_adjusted sg_data_point meter^-1 steradian^-1 total volume red scattering coefficient using manufacturer-supplied dark counts and scaling factor
CTD_qc string Whether to trust the corrected CTD values
GPS1_qc string Whether to trust the GPS1 information
GPS2_qc string Whether to trust the GPS2 information
GPSE_qc string Whether to trust the final GPS information
aa4330 string
aanderaa4330_qc string Whether to trust the Aanderaa 3880 results
avg_latitude string degrees_north The average latitude of the dive
depth_avg_curr_east string m/s Eastward component of the dive depth-average current based on hdm
depth_avg_curr_east_gsm string m/s Eastward component of depth-average current based on gsm
depth_avg_curr_north string m/s Northward component of the dive depth-average current based on hdm
depth_avg_curr_north_gsm string m/s Northward component of depth-average current based on gsm
depth_avg_curr_qc string Whether to trust the dive depth-average current values and displacements
flight_avg_speed_east string m/s Eastward component of flight average speed based on hdm
flight_avg_speed_east_gsm string m/s Eastward component of flight average speed based on gsm
flight_avg_speed_north string m/s Northward component of flight average speed based on hdm
flight_avg_speed_north_gsm string m/s Northward component of flight average speed based on gsm
glider string
hdm_qc string Whether corrected temperatures, salinities, and velocities from the hydrodynamic model converged on a consistent solution
latlong_qc string Whether to trust the dive estimated latitude and longitude estimates
log_10V_AH string
log_24V_AH string
log_AD7714Ch0Gain string
log_AH0_10V string
log_AH0_24V string
log_ALTIM_BOTTOM_PING_RANGE string
log_ALTIM_BOTTOM_TURN_MARGIN string
log_ALTIM_FREQUENCY string
log_ALTIM_PING_DELTA string
log_ALTIM_PING_DEPTH string
log_ALTIM_PULSE string
log_ALTIM_SENSITIVITY string
log_ALTIM_TOP_MIN_OBSTACLE string
log_ALTIM_TOP_PING_RANGE string
log_ALTIM_TOP_TURN_MARGIN string
log_APOGEE_PITCH string
log_CALL_NDIVES string
log_CALL_TRIES string
log_CALL_WAIT string
log_CAPMAXSIZE string
log_CAPUPLOAD string
log_CAP_FILE_SIZE string
log_CF8_MAXERRORS string
log_CFSIZE string
log_COMM_SEQ string
log_COMPASS2_DEVICE string
log_COMPASS_DEVICE string
log_COMPASS_USE string
log_COURSE_BIAS string
log_C_PITCH string
log_C_ROLL_CLIMB string
log_C_ROLL_DIVE string
log_C_VBD string
log_DATA_FILE_SIZE string
log_DBDW string
log_DEEPGLIDER string
log_DEVICE1 string
log_DEVICE2 string
log_DEVICE3 string
log_DEVICE4 string
log_DEVICE5 string
log_DEVICE6 string
log_DEVICES string
log_DEVICE_MAMPS string
log_DEVICE_SECS string
log_DIVE string
log_D_ABORT string
log_D_BOOST string
log_D_CALL string
log_D_FINISH string
log_D_FLARE string
log_D_GRID string
log_D_NO_BLEED string
log_D_OFFGRID string
log_D_PITCH string
log_D_SAFE string
log_D_SURF string
log_D_TGT string
log_ERRORS string
log_ESCAPE_HEADING string
log_ESCAPE_HEADING_DELTA string
log_FERRY_MAX string
log_FG_AHR_10V string
log_FG_AHR_10Vo string
log_FG_AHR_24V string
log_FG_AHR_24Vo string
log_FILEMGR string
log_FIX_MISSING_TIMEOUT string
log_GLIDE_SLOPE string
log_GPS string String reported in logfile for GPS fix (first surface position after dive)
log_GPS1 string String reported in logfile for GPS1 fix (first surface position before dive)
log_GPS2 string String reported in logfile for GPS2 fix (last surface position before dive)
log_GPS_DEVICE string
log_HD_A string Hydrodynamic lift factor for given hull shape (1/degrees of attack angle)
log_HD_B string Hydrodynamic drag factor for given hull shape (Pa^(-1/4))
log_HD_C string Hydrodynamic induced drag factor for given hull shape (1/degrees^2 of attack angle)
log_HEADING string
log_HEAD_ERRBAND string
log_HEAPDBG string
log_HUMID string
log_ICE_FREEZE_MARGIN string
log_ID string
log_INTERNAL_PRESSURE string
log_INT_PRESSURE_SLOPE string
log_INT_PRESSURE_YINT string
log_IRIDIUM_FIX string
log_KALMAN_CONTROL string
log_KALMAN_USE string
log_KALMAN_X string
log_KALMAN_Y string
log_LOGGERDEVICE1 string
log_LOGGERDEVICE2 string
log_LOGGERDEVICE3 string
log_LOGGERDEVICE4 string
log_LOGGERS string
log_MASS string
log_MASS_COMP string
log_MAXI_10V string
log_MAXI_24V string
log_MAX_BUOY string
log_MEM string
log_MHEAD_RNG_PITCHd_Wd string
log_MINV_10V string
log_MINV_24V string
log_MISSION string
log_MOTHERBOARD string
log_NAV_MODE string
log_NOCOMM_ACTION string
log_N_DIVES string
log_N_FILEKB string
log_N_GPS string
log_N_NOCOMM string
log_N_NOSURFACE string
log_PHONE_DEVICE string
log_PHONE_SUPPLY string
log_PITCH_ADJ_DBAND string
log_PITCH_ADJ_GAIN string
log_PITCH_AD_RATE string
log_PITCH_CNV string
log_PITCH_DBAND string
log_PITCH_GAIN string
log_PITCH_MAX string
log_PITCH_MAXERRORS string
log_PITCH_MIN string
log_PITCH_TIMEOUT string
log_PITCH_VBD_SHIFT string
log_PITCH_W_DBAND string
log_PITCH_W_GAIN string
log_PRESSURE_SLOPE string
log_PRESSURE_YINT string
log_PROTOCOL string
log_P_OVSHOOT string
log_P_OVSHOOT_WITHG string
log_RAFOS_CORR_THRESH string
log_RAFOS_DEVICE string
log_RAFOS_HIT_WINDOW string
log_RAFOS_MMODEM string
log_RAFOS_PEAK_OFFSET string
log_RELAUNCH string
log_RHO string gram/cc Expected density at deepest point over the deployment
log_ROLL_ADJ_DBAND string
log_ROLL_ADJ_GAIN string
log_ROLL_AD_RATE string
log_ROLL_CNV string
log_ROLL_DEG string
log_ROLL_MAX string
log_ROLL_MAXERRORS string
log_ROLL_MIN string
log_ROLL_TIMEOUT string
log_R_PORT_OVSHOOT string
log_R_STBD_OVSHOOT string
log_SEABIRD_C_G string
log_SEABIRD_C_H string
log_SEABIRD_C_I string
log_SEABIRD_C_J string
log_SEABIRD_C_Z string Conductivity zero frequency
log_SEABIRD_T_G string
log_SEABIRD_T_H string
log_SEABIRD_T_I string
log_SEABIRD_T_J string
log_SENSORS string
log_SENSOR_MAMPS string
log_SENSOR_SECS string
log_SIM_PITCH string
log_SIM_W string
log_SM_CC string
log_SM_CCo string
log_SM_GC string
log_SPEED_FACTOR string
log_SPEED_LIMITS string
log_STOP_T string
log_STROBE string
log_SURFACE_URGENCY string
log_SURFACE_URGENCY_FORCE string
log_SURFACE_URGENCY_TRY string
log_TCM_PITCH_OFFSET string
log_TCM_ROLL_OFFSET string
log_TCM_TEMP string
log_TGT_AUTO_DEFAULT string
log_TGT_DEFAULT_LAT string
log_TGT_DEFAULT_LON string
log_TGT_LATLONG string
log_TGT_NAME string
log_TGT_RADIUS string
log_TT8_MAMPS string
log_T_ABORT string
log_T_BOOST string
log_T_DIVE string
log_T_EPIRB string
log_T_GPS string
log_T_LOITER string
log_T_MISSION string
log_T_NO_W string
log_T_RSLEEP string
log_T_SLOITER string
log_T_TURN string
log_T_TURN_SAMPINT string
log_T_WATCHDOG string
log_UNCOM_BLEED string
log_UPLOAD_DIVES_MAX string
log_USE_BATHY string
log_USE_ICE string
log_VBD_BLEED_AD_RATE string
log_VBD_CNV string
log_VBD_DBAND string
log_VBD_LP_IGNORE string
log_VBD_MAX string
log_VBD_MAXERRORS string
log_VBD_MIN string
log_VBD_PUMP_AD_RATE_APOGEE string
log_VBD_PUMP_AD_RATE_SURFACE string
log_VBD_TIMEOUT string
log_W_ADJ_DBAND string
log_XPDR_DEVICE string
log_XPDR_INHIBIT string
log_XPDR_PINGS string
log_XPDR_VALID string
log__CALLS string
log__SM_ANGLEo string
log__SM_DEPTHo string
log__XMS_NAKs string
log__XMS_TOUTs string
magnetic_variation string The magnetic variance from true north (degrees)
magnetometer string
reviewed string Whether a scientist has reviewed and approved this profile
sat1000 string
sat1001 string
sbe41 string
sg_cal_abs_compress string cc/dbar SG vehicle compressibility
sg_cal_c_g string
sg_cal_c_h string
sg_cal_c_i string
sg_cal_c_j string
sg_cal_calibcomm string
sg_cal_calibcomm_optode string
sg_cal_calibcomm_wetlabs string
sg_cal_comm_oxy_type string
sg_cal_cpcor string Nominal compression factor of conductivity tube with pressure
sg_cal_ctcor string Nominal thermal expansion factor of a cube of boro-silicate glass
sg_cal_hd_a string Hydrodynamic lift factor for given hull shape (1/degrees of attack angle)
sg_cal_hd_b string Hydrodynamic drag factor for given hull shape (Pa^(-1/4))
sg_cal_hd_c string Hydrodynamic induced drag factor for given hull shape (1/radians^2 of attack angle)
sg_cal_id_str string Three digit vehicle identification string
sg_cal_mass string kg Mass of the glider
sg_cal_mass_comp string kg Mass of the compressee
sg_cal_mission_title string Description of mission
sg_cal_optode_ConcCoef0 string
sg_cal_optode_ConcCoef1 string
sg_cal_optode_FoilCoefA0 string
sg_cal_optode_FoilCoefA1 string
sg_cal_optode_FoilCoefA10 string
sg_cal_optode_FoilCoefA11 string
sg_cal_optode_FoilCoefA12 string
sg_cal_optode_FoilCoefA13 string
sg_cal_optode_FoilCoefA2 string
sg_cal_optode_FoilCoefA3 string
sg_cal_optode_FoilCoefA4 string
sg_cal_optode_FoilCoefA5 string
sg_cal_optode_FoilCoefA6 string
sg_cal_optode_FoilCoefA7 string
sg_cal_optode_FoilCoefA8 string
sg_cal_optode_FoilCoefA9 string
sg_cal_optode_FoilCoefB0 string
sg_cal_optode_FoilCoefB1 string
sg_cal_optode_FoilCoefB10 string
sg_cal_optode_FoilCoefB11 string
sg_cal_optode_FoilCoefB12 string
sg_cal_optode_FoilCoefB13 string
sg_cal_optode_FoilCoefB2 string
sg_cal_optode_FoilCoefB3 string
sg_cal_optode_FoilCoefB4 string
sg_cal_optode_FoilCoefB5 string
sg_cal_optode_FoilCoefB6 string
sg_cal_optode_FoilCoefB7 string
sg_cal_optode_FoilCoefB8 string
sg_cal_optode_FoilCoefB9 string
sg_cal_optode_PhaseCoef0 string
sg_cal_optode_PhaseCoef1 string
sg_cal_optode_PhaseCoef2 string
sg_cal_optode_PhaseCoef3 string
sg_cal_optode_SVUCoef0 string
sg_cal_optode_SVUCoef1 string
sg_cal_optode_SVUCoef2 string
sg_cal_optode_SVUCoef3 string
sg_cal_optode_SVUCoef4 string
sg_cal_optode_SVUCoef5 string
sg_cal_optode_SVUCoef6 string
sg_cal_optode_SVU_enabled string
sg_cal_rho0 string kg/m^3 Typical expected density of seawater for this deployment
sg_cal_sbect_modes string Number of modes to use for thermal-inertia correction
sg_cal_t_g string
sg_cal_t_h string
sg_cal_t_i string
sg_cal_t_j string
sg_cal_therm_expan string cc/degrees_Celsius SG thermal expansion coeff
sg_cal_volmax string m^3 Maximum displaced volume of the glider
sg_cal_wlbb2fl_sig470nm_dark_counts string
sg_cal_wlbb2fl_sig470nm_resolution_counts string
sg_cal_wlbb2fl_sig470nm_scale_factor string
sg_cal_wlbb2fl_sig695nm_dark_counts string
sg_cal_wlbb2fl_sig695nm_resolution_counts string
sg_cal_wlbb2fl_sig695nm_scale_factor string
sg_cal_wlbb2fl_sig700nm_dark_counts string
sg_cal_wlbb2fl_sig700nm_resolution_counts string
sg_cal_wlbb2fl_sig700nm_scale_factor string
start_of_climb_time string seconds Elapsed seconds after dive start when second (positive) apogee pump starts
surface_curr_east string cm/s Eastward component of surface current
surface_curr_north string cm/s Northward component of surface current
surface_curr_qc string Whether to trust the surface current values
wlbb2fl string
trajectory trajectory Dive number for observations 
names = [
    'ctd_depth',
    'ctd_time',
    'ctd_pressure',
    'salinity',
    'salinity_raw',
    'temperature',
    'temperature_raw',
    'temperature_qc',
    'eng_wlbb2fl_sig470nm',
    'eng_wlbb2fl_sig695nm',
    'eng_wlbb2fl_sig700nm'
]

ds_dict = gt.load.seaglider_basestation_netCDFs(filenames, names, return_merged=True, keep_global_attrs=False)

DIMENSION: sg_data_point
{latitude, longitude, temperature, ctd_pressure, salinity_raw, temperature_qc, ctd_time, salinity, ctd_depth, temperature_raw}

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2566/2566 [00:19<00:00, 129.19it/s]

Cannot merge data - not enough time indexed DataFrames
Returning unmerged dataframes

some file lack enough information/details (or are incomplete dives):

  • early dive issues (2??)

  • p401067[34]

ds_dict['sg_data_point']

<xarray.Dataset>
Dimensions:          (sg_data_point: 349365)
Coordinates:
    latitude         (sg_data_point) float64 61.48 61.48 61.48 ... 57.93 57.93
    longitude        (sg_data_point) float64 -173.8 -173.8 ... -169.1 -169.1
    ctd_depth        (sg_data_point) float64 0.7374 0.2013 ... 1.294 -0.1832
    ctd_time_dt64    (sg_data_point) datetime64[ns] 2017-08-13T21:07:40 ... 2...
Dimensions without coordinates: sg_data_point
Data variables:
    temperature      (sg_data_point) float64 9.512 9.513 9.511 ... 7.004 7.024
    ctd_pressure     (sg_data_point) float64 0.7451 0.2038 ... 1.306 -0.1844
    salinity_raw     (sg_data_point) float64 31.16 31.15 31.15 ... 31.46 2.506
    temperature_qc   (sg_data_point) int64 1 1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1
    ctd_time         (sg_data_point) float64 1.503e+09 1.503e+09 ... 1.506e+09
    salinity         (sg_data_point) float64 nan nan nan nan ... 31.46 nan nan
    temperature_raw  (sg_data_point) float64 9.512 9.513 9.511 ... 7.004 7.023
    dives            (sg_data_point) float64 1e+03 1e+03 ... 3.612e+03 3.612e+03
Attributes:
    date_created:             2021-07-26 13:55:28
    number_of_dives:          3612.0
    files:                    ['p4011000.nc', 'p4011001.nc', 'p4011002.nc', '...
    time_coverage_start:      2017-08-13 21:07:40
    time_coverage_end:        2017-09-26 16:05:42
    geospatial_vertical_min:  -0.5030842835078432
    geospatial_vertical_max:  67.57316616647842
    geospatial_lat_min:       57.89913235353872
    geospatial_lat_max:       62.14208166934556
    geospatial_lon_min:       -174.5055983333333
    geospatial_lon_max:       -168.8639759711956
    processing:               [2021-07-26 13:55:28] imported data with Glider...
dat = ds_dict['sg_data_point'].rename({
    'ctd_pressure': 'pressure',
    'ctd_depth': 'depth',
    'ctd_time_dt64': 'time',
    'eng_wlbb2fl_sig695nm': 'flr_raw'
})


# name coordinates for quicker plotting
x = dat.dives
y = dat.depth

gt.plot(x, y, dat.temperature_raw, shading='nearest', cmap=cmo.thermal, robust=True)
title('Original Data')
show()
../../../_images/EcoFOCI_GliderTools_Readme_26_0.png 
gt.plot(x, y, dat.temperature, shading='nearest', cmap=cmo.thermal, robust=True)
title('Original Data - UW QC')
show()
../../../_images/EcoFOCI_GliderTools_Readme_27_0.png 
#lots of options available, but lets filter (without much smoothing if any at all, lets do it in the verticle for sure and contemplate horizonatl filtering too)

temp_qc = gt.calc_physics(dat.temperature, x, y, 
                          iqr=1.5, depth_threshold=0, #<-- depth 
                          spike_window=5, spike_method='median',
                          savitzky_golay_window=0, savitzky_golay_order=2) #<--challenged by sharp interface, made it clearly wrong with window=11, order=2

# PLOTTING
fig, ax = plt.subplots(3, 1, figsize=[9, 8.5], sharex=True, dpi=90)

gt.plot(x, y, dat.temperature, cmap=cmo.thermal, ax=ax[0])
gt.plot(x, y, temp_qc, cmap=cmo.thermal, ax=ax[1])
gt.plot(x, y, temp_qc - dat.temperature, cmap=cm.RdBu_r, vmin=-0.05, vmax=0.05, ax=ax[2])

[a.set_xlabel('') for a in ax]

ax[0].cb.set_label('Original Data')
ax[1].cb.set_label('Cleaned Data')
ax[2].cb.set_label('Difference from Original')

plt.show()

==================================================
Physics Variable:
	Removing outliers with IQR * 1.5: 0 obs
	Removing spikes with rolling median (spike window=5)
../../../_images/EcoFOCI_GliderTools_Readme_28_1.png 
#lots of options available, but lets filter (without much smoothing if any at all, lets do it in the verticle for sure and contemplate horizonatl filtering too)
# how does it do for despiking at interface?

salt_qc = gt.calc_physics(dat.salinity, x, y, 
                          mask_frac=0.2, iqr=2.5, 
                          spike_window=5, spike_method='median', 
                          savitzky_golay_window=0, savitzky_golay_order=2)
        
# PLOTTING
fig, ax = plt.subplots(3, 1, figsize=[9, 8.5], sharex=True, dpi=90)

gt.plot(x, y, dat.salinity, cmap=cmo.haline, ax=ax[0])
gt.plot(x, y, salt_qc, cmap=cmo.haline, ax=ax[1])
gt.plot(x, y, salt_qc - dat.salinity, cmap=cm.RdBu_r, vmin=-0.02, vmax=0.02, ax=ax[2])

[a.set_xlabel('') for a in ax]

ax[0].cb.set_label('Original Data')
ax[1].cb.set_label('Cleaned Data')
ax[2].cb.set_label('Difference from Original')

plt.show()

==================================================
Physics Variable:
	Removing outliers with IQR * 2.5: 313 obs
	Removing spikes with rolling median (spike window=5)
	Removing horizontal outliers (fraction=0.2, multiplier=2.5)
../../../_images/EcoFOCI_GliderTools_Readme_29_1.png

Insert data into original dataset

dat['temp_gt_qc'] = temp_qc
dat['salt_gt_qc'] = salt_qc

Derived Variables

dens0 = gt.physics.potential_density(salt_qc, temp_qc, dat.pressure, dat.latitude, dat.longitude)

mld = gt.physics.mixed_layer_depth(x, y, dens0)
mld_smoothed = mld.rolling(10, min_periods=3).mean()

mld_mask = gt.physics.mixed_layer_depth(x, y, dens0, return_as_mask=True)
mld_grid = gt.grid_data(x, y, mld_mask, verbose=False)

fig, ax = plt.subplots(1, 2, figsize=[9, 3], dpi=100, sharey=True)

mld_smoothed.plot(ax=ax[0])
gt.plot(mld_grid, ax=ax[1])

[a.set_ylim(100, 0) for a in ax]

ax[0].set_ylabel('Depth (m)')
[a.set_xlabel('Dives') for a in ax]
xticks(rotation=0)

fig.tight_layout()
../../../_images/EcoFOCI_GliderTools_Readme_35_0.png

Ancillary Instrumentation: Backscatter/Fluorometry , Oxygen , PAR

  • apply calibration curves from factor

  • despike in vertical

  • quenching and other fluorometric adjustments

  • oxygen equilibration in ML boundary

Gridding and interpolation

Vertical gridding

It is often more convenient and computationally efficient to work with data that has been gridded to a standard vertical grid (i.e. depths have been binned). GliderTools offers very easy to use and efficient tools to grid data once all the processing has been completed.

The first task is to select the bin size of the data that will be gridded. GliderTools automatically selects bin sizes according to the sampling frequency of the dataset for every 50m. This is shown in the figure below, where the 2D histogram shows the sampling frequency (by depth) and the line shows the automatically selected bin size rounded up to the nearest 0.5m.

ax = gt.plot.bin_size(dat.depth, cmap=mpl.cm.Blues)
ax.set_xlim(0, 6)
line = ax.get_children()[1]
line.set_linewidth(6)
line.set_color('orange')

legend = ax.get_children()[-2]
legend.set_visible(False)
../../../_images/EcoFOCI_GliderTools_Readme_39_0.png

1m seems the optimal gridding for the 2017 SG missions

previous

Oculus Glider Notebooks

next

MetaData Rediscovery