sherlock.imports.ned module

Import ned stream into sherlock-catalogues database

Author

David Young

class ned(log, settings=False, pathToDataFile=False, version=False, catalogueName='', coordinateList=[], radiusArcsec=False)[source]

Bases: sherlock.imports._base_importer._base_importer

Using a list of coordinates, query the online NED database and import sources found within a given search radius of each of the loctions into the sherlock-catalogues database

The code:

1. Uses the list of transient coordinates and queries NED (conesearch) for the results within the given search radius
2. Creates the `tcs_cat_ned_stream` table if it doesn't exist
3. Adds the resulting matched NED IDs/Names to the `tcs_cat_ned_stream` table
4. Updates the NED query history table
5. Queris NED via NED IDs (object search) for the remaining source metadata to be added to the `tcs_cat_ned_stream` table

Note it’s up to the user to filter the input coordinate list by checking whether or not the same area of the sky has been imported into the tcs_cat_ned_stream table recently (by checking the tcs_helper_ned_query_history table)

Key Arguments

  • dbConn – mysql database connection

  • log – logger

  • settings – the settings dictionary

  • coordinateList – list of coordinates (a list of strings with RA and DEC space separated)

  • radiusArcsec - - the radius in arcsec with which to perform the initial NED conesearch. Default * False*

Usage

To import the ned catalogue stream, run the following:

from sherlock.imports import ned
stream = ned(
    log=log,
    settings=settings,
    coordinateList=["23.12323 -12.34343","345.43234 45.26789"],
    radiusArcsec=180
)
stream.ingest()
- test this code is still working after changes
- add option to filter coordinate list via the `tcs_helper_ned_query_history` table
- check sublime snippet exists
- clip any useful text to docs mindmap
ingest()[source]

Perform conesearches of the online NED database and import the results into a the sherlock-database

The code:

1. uses the list of transient coordinates and queries NED for the results within the given search radius
2. Creates the `tcs_cat_ned_stream` table if it doesn't exist
3. Adds the resulting NED IDs/Names to the `tcs_cat_ned_stream` table
4. Updates the NED query history table
5. Queris NED via NED IDs for the remaining source metadata to be added to the `tcs_cat_ned_stream` table

Usage

Having setup the NED object with a coordinate list and cone-search radius, run the ingest() method

stream.ingest()
- check sublime snippet exists
- clip any useful text to docs mindmap
- regenerate the docs and check redendering of this docstring
_create_dictionary_of_ned()[source]

Create a list of dictionaries containing all the object ids (NED names) in the ned stream

Return

  • dictList - a list of dictionaries containing all the object ids (NED names) in the ned stream

Usage

dictList = stream._create_dictionary_of_ned()
_update_ned_query_history()[source]

Update the database helper table to give details of the ned cone searches performed

Usage:

```python
stream._update_ned_query_history()
```
_download_ned_source_metadata()[source]

Query NED using the names of the NED sources in our local database to retrieve extra metadata

Usage:

```python
stream._download_ned_source_metadata()
```
_get_ned_sources_needing_metadata()[source]

Get the names of 50000 or less NED sources that still require metabase in the database

Return

  • len(self.theseIds) – the number of NED IDs returned

Usage:

```python
numberSources = stream._get_ned_sources_needing_metadata()
```
_do_ned_namesearch_queries_and_add_resulting_metadata_to_database(batchCount)[source]

Query NED via name searcha and add result metadata to database

Key Arguments

  • batchCount - the index number of the batch sent to NED (only needed for printing to STDOUT to give user idea of progress)

Usage:

```python
numberSources = stream._do_ned_namesearch_queries_and_add_resulting_metadata_to_database(batchCount=10)
```
_count_ned_sources_in_database_requiring_metadata()[source]

Count the sources in the NED table requiring metadata

Return

  • self.total, self.batches – total number of galaxies needing metadata & the number of batches required to be sent to NED

Usage:

```python
totalRemaining, numberOfBatches = stream._count_ned_sources_in_database_requiring_metadata()
```
_add_htmids_to_database_table()[source]

Add HTMIDs to database table once all the data has been imported (HTM Levels 10,13,16)

Usage

self._add_htmids_to_database_table()
_update_database_helper_table()[source]

Update the sherlock catalogues database helper table with the time-stamp of when this catlogue was last updated

Usage

self._update_database_helper_table()
add_data_to_database_table(dictList, createStatement=False)[source]

Import data in the list of dictionaries in the requested database table

Also adds HTMIDs and updates the sherlock-catalogue database helper table with the time-stamp of when the imported catlogue was last updated

Key Arguments

  • dictList - a list of dictionaries containing all the rows in the catalogue to be imported

  • createStatement - the table’s mysql create statement (used to generate table if it does not yet exist in database). Default False

Usage

self.add_data_to_database_table(
    dictList=dictList,
    createStatement=createStatement
)
- Write a checklist for creating a new sherlock database importer