sherlock ∞

mine a library of historical and on-going astronomical survey data in an attempt to identify the sources of transient/variable events, and predict their classifications based on crossmatched data.

Documentation for sherlock is hosted by Read the Docs ( development version and master version). The code lives on github. Please report any issues you find here.

Features ∞

Installation ∞

The easiest way to install sherlock is to use pip (here we show the install inside of a conda environment):

conda create -n sherlock python=3.7 pip
conda activate sherlock
pip install sherlock

Or you can clone the github repo and install from a local version of the code:

git clone git@github.com:thespacedoctor/sherlock.git
cd sherlock
python setup.py install

To upgrade to the latest version of sherlock use the command:

pip install sherlock --upgrade

To check installation was successful run sherlock -v. This should return the version number of the install.

Development ∞

If you want to tinker with the code, then install in development mode. This means you can modify the code from your cloned repo:

git clone git@github.com:thespacedoctor/sherlock.git
cd sherlock
python setup.py develop

Pull requests are welcomed!

Initialisation ∞

Before using sherlock you need to use the init command to generate a user settings file. Running the following creates a yaml settings file in your home folder under ~/.config/sherlock/sherlock.yaml:

sherlock init

The file is initially populated with sherlock’s default settings which can be adjusted to your preference.

If at any point the user settings file becomes corrupted or you just want to start afresh, simply trash the sherlock.yaml file and rerun sherlock init.

Modifying the Settings ∞

Once created, open the settings file in any text editor and make any modifications needed.

Basic Python Setup ∞

If you plan to use sherlock in your own scripts you will first need to parse your settings file and set up logging etc. One quick way to do this is to use the fundamentals package to give you a logger, a settings dictionary and a database connection (if connection details given in settings file):

## SOME BASIC SETUP FOR LOGGING, SETTINGS ETC
from fundamentals import tools
from os.path import expanduser
home = expanduser("~")
settingsFile  = home + "/.config/sherlock/sherlock.yaml"
su = tools(
    arguments={"settingsFile": settingsFile},
    docString=__doc__,
)
arguments, settings, log, dbConn = su.setup()

Todo List ∞

Todo

update the package tutorial if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/catalogue_conesearch.py:docstring of sherlock.catalogue_conesearch.catalogue_conesearch, line 25.)

Todo

write a checklist for adding a new catalogue to the sherlock database and reference it from here (use the image below of the tcs_helper_catalogue_views_info table)

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/commonutils/get_crossmatch_catalogues_column_map.py:docstring of sherlock.commonutils.get_crossmatch_catalogues_column_map.get_crossmatch_catalogues_column_map, line 7.)

Todo

update the package tutorial if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier, line 26.)

Todo

add usage info
create a sublime snippet for usage
update package tutorial if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier._consolidate_coordinateList, line 16.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier.classification_annotations, line 16.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier.update_classification_annotations_and_summaries, line 19.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier.update_peak_magnitudes, line 16.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier._create_tables_if_not_exist, line 16.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier.generate_match_annotation, line 25.)

Todo

update the package tutorial if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/catalogue_conesearch.py:docstring of sherlock.catalogue_conesearch.catalogue_conesearch, line 25.)

Todo

write a checklist for adding a new catalogue to the sherlock database and reference it from here (use the image below of the tcs_helper_catalogue_views_info table)

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/commonutils/get_crossmatch_catalogues_column_map.py:docstring of sherlock.commonutils.get_crossmatch_catalogues_column_map.get_crossmatch_catalogues_column_map, line 7.)

Todo

add an entry in the tutorial to clean database tables

```python from sherlock.database_cleaner import database_cleaner db = database_cleaner(

log=log, settings=settings

) db.clean() ```

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/database_cleaner.py:docstring of sherlock.database_cleaner.database_cleaner, line 15.)

Todo

update the package tutorial if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier, line 26.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier.classification_annotations, line 16.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier.generate_match_annotation, line 25.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier.update_classification_annotations_and_summaries, line 19.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/transient_classifier.py:docstring of sherlock.transient_classifier.transient_classifier.update_peak_magnitudes, line 16.)

Todo

docuument cl_utils module
tidy usage text

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/cl_utils.py:docstring of sherlock.cl_utils, line 7.)

Todo

add an entry in the tutorial to clean database tables

```python from sherlock.database_cleaner import database_cleaner db = database_cleaner(

log=log, settings=settings

) db.clean() ```

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/database_cleaner.py:docstring of sherlock.database_cleaner.database_cleaner, line 15.)

Todo

add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/envs/develop/lib/python3.7/site-packages/qub_sherlock-2.2.0-py3.7.egg/sherlock/database_cleaner.py:docstring of sherlock.database_cleaner.database_cleaner._create_tcs_help_tables, line 25.)

Todo

nice!

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/qub-sherlock/checkouts/develop/docs/source/_template_.md, line 1.)

Release Notes ∞

v2.2.0 - August 2, 2021

FEATURE: Default search algorithm now version-controlled and ship alongside code. Users can override the default algorithm if they need to.
FEATURE: Hidden database-batch-size and cpu-pool-size settings added - power users can access and change in settings files.
ENHANCEMENT: Speed improvements (5-10 times processing speed increase).
REFACTOR/FIX: synonym match break now removed by default so a location now gets matched against all catalogues regardless of whether or not a synonym match is initially found (e.g. right on top of a catalogued star). Resolves edge cases where true SNe are mis-classified as VS as they are located just too close to a stellar source, or galaxy source mis-identified as stellar in one catalogue but correctly as galaxy in another.
FIX: some URLs to NED objects, found in the human readable Sherlock annotations, that were not resolving now do.

v2.1.8 - April 16, 2021#bugFixFriday

REFACTOR: catalogue quality weights now used within the merged source sets to sort the data. This fixed some mis-matched annotation issues.
REFACTOR: updating db connection settings to use QUB jumpbox.
REFACTOR/FIX Reduced associated source merging radius from 3 arcsec to 1 arcsec (1/9th of the original matching area). This seems to correct many of the cases in which 2 or more distinct catalogued sources were getting merged/blended into one association. This resulted in in-correct classification prediction and wrongly associated distances.
FIXED: annotation mismatches. If transient is classified as a SN then the underlying sources is identified as a galaxy in the annotation.

v2.1.7 - September 29, 2020

FIXED: lite version introduced a couple of small bugs. Fixed.

v2.1.6 - September 7, 2020

ENHANCEMENT: transient human-readable annotations added to the classifications dictionary return of the transient_classifier.classify() method.
ENHANCEMENT: a lite parameter has been added to the Sherlock transient_classifier object. When set to True only top-ranked crossmatches are returned with a limited column set.
ENHANCEMENT: snapshot annotation is now added into the classification dictionary return. The classification dictionary is now {'MyTransientId01': ['Type', 'Annotation'], 'MyTransientId02': ['Type', 'Annotation'],}
REFACTOR: small change to query that writes NED source magnitudes that vastly improves speed
FIXED: small issue where a missing error in photoz was causing annotations to not complete

v2.1.5 - June 22, 2020

refactor: tunnel changes due to new jump box installation at QUB
fixed: some NED galaxies not reported if redshift does not exist but a semi-major axis is given

v2.1.4 - June 18, 2020

fixed: an empty crossmatch set could cause a transient database update when not required

v2.1.3 - June 5, 2020

refactor: stop sherlock checking for transient database triggers if running in non-update mode
fixed: another sdss photoz issue where photoz ranking about specz

v2.1.2 - May 24, 2020

fixed: merged result parameters are now merged correctly

v2.1.1 - May 21, 2020

fixed: fixes to cl utils

v2.1.0 - May 20, 2020

Now compatible with Python 3.*

API Reference ∞

Modules ∞

`sherlock.commonutils`	common tools used throughout package
`sherlock.imports`
`sherlock.utKit`	Unit testing tools

Classes ∞

`sherlock.catalogue_conesearch`	The worker class for the conesearch module
`sherlock.commonutils.update_wiki_pages`	Update sherlock’s github wiki pages with some useful info regarding the crossmatch database catalogue tables
`sherlock.database`	the database object for sherlock, setting up ssh tunnels and various database connections
`sherlock.database_cleaner`	Clean and maintain the database helper tables used by sherlock
`sherlock.imports.ifs`	Importer for the Multi Unit Spectroscopic Explorer (MUSE) IFS galaxy catalogue stream
`sherlock.imports.ned`	Using a list of coordinates, query the online NED database and import sources found within a given search radius of each of the loctions into the sherlock-catalogues database
`sherlock.imports.ned_d`	Import the `NED-D <https://ned.ipac.caltech.edu/Library/Distances/>`_ galaxy catalogue in to the sherlock-catalogues database
`sherlock.imports.veron`	importer object for the VERON AGN catalogue
`sherlock.transient_catalogue_crossmatch`	crossmatch a list of transients against a suite of catalogues according to given search algorithm
`sherlock.transient_classifier`	The Sherlock Transient Classifier

Functions ∞

`sherlock.commonutils.get_crossmatch_catalogues_column_map`	Query the sherlock-catalogues helper tables to generate a map of the important columns of each catalogue
`sherlock.commonutils.getpackagepath`	Get the root path for this python package - used in unit testing code

A-Z Index ∞

Modules

`sherlock.commonutils`	common tools used throughout package
`sherlock.imports`
`sherlock.utKit`	Unit testing tools

Classes

`sherlock.catalogue_conesearch`	The worker class for the conesearch module
`sherlock.commonutils.update_wiki_pages`	Update sherlock’s github wiki pages with some useful info regarding the crossmatch database catalogue tables
`sherlock.database`	the database object for sherlock, setting up ssh tunnels and various database connections
`sherlock.database_cleaner`	Clean and maintain the database helper tables used by sherlock
`sherlock.imports.ifs`	Importer for the Multi Unit Spectroscopic Explorer (MUSE) IFS galaxy catalogue stream
`sherlock.imports.ned`	Using a list of coordinates, query the online NED database and import sources found within a given search radius of each of the loctions into the sherlock-catalogues database
`sherlock.imports.ned_d`	Import the `NED-D <https://ned.ipac.caltech.edu/Library/Distances/>`_ galaxy catalogue in to the sherlock-catalogues database
`sherlock.imports.veron`	importer object for the VERON AGN catalogue
`sherlock.transient_catalogue_crossmatch`	crossmatch a list of transients against a suite of catalogues according to given search algorithm
`sherlock.transient_classifier`	The Sherlock Transient Classifier

Functions

`sherlock.commonutils.get_crossmatch_catalogues_column_map`	Query the sherlock-catalogues helper tables to generate a map of the important columns of each catalogue
`sherlock.commonutils.getpackagepath`	Get the root path for this python package - used in unit testing code

Release Notes ∞

v2.2.0 - August 2, 2021

FEATURE: Default search algorithm now version-controlled and ship alongside code. Users can override the default algorithm if they need to.
FEATURE: Hidden database-batch-size and cpu-pool-size settings added - power users can access and change in settings files.
ENHANCEMENT: Speed improvements (5-10 times processing speed increase).
REFACTOR/FIX: synonym match break now removed by default so a location now gets matched against all catalogues regardless of whether or not a synonym match is initially found (e.g. right on top of a catalogued star). Resolves edge cases where true SNe are mis-classified as VS as they are located just too close to a stellar source, or galaxy source mis-identified as stellar in one catalogue but correctly as galaxy in another.
FIX: some URLs to NED objects, found in the human readable Sherlock annotations, that were not resolving now do.

v2.1.8 - April 16, 2021#bugFixFriday

REFACTOR: catalogue quality weights now used within the merged source sets to sort the data. This fixed some mis-matched annotation issues.
REFACTOR: updating db connection settings to use QUB jumpbox.
REFACTOR/FIX Reduced associated source merging radius from 3 arcsec to 1 arcsec (1/9th of the original matching area). This seems to correct many of the cases in which 2 or more distinct catalogued sources were getting merged/blended into one association. This resulted in in-correct classification prediction and wrongly associated distances.
FIXED: annotation mismatches. If transient is classified as a SN then the underlying sources is identified as a galaxy in the annotation.

v2.1.7 - September 29, 2020

FIXED: lite version introduced a couple of small bugs. Fixed.

v2.1.6 - September 7, 2020

ENHANCEMENT: transient human-readable annotations added to the classifications dictionary return of the transient_classifier.classify() method.
ENHANCEMENT: a lite parameter has been added to the Sherlock transient_classifier object. When set to True only top-ranked crossmatches are returned with a limited column set.
ENHANCEMENT: snapshot annotation is now added into the classification dictionary return. The classification dictionary is now {'MyTransientId01': ['Type', 'Annotation'], 'MyTransientId02': ['Type', 'Annotation'],}
REFACTOR: small change to query that writes NED source magnitudes that vastly improves speed
FIXED: small issue where a missing error in photoz was causing annotations to not complete

v2.1.5 - June 22, 2020

refactor: tunnel changes due to new jump box installation at QUB
fixed: some NED galaxies not reported if redshift does not exist but a semi-major axis is given

v2.1.4 - June 18, 2020

fixed: an empty crossmatch set could cause a transient database update when not required

v2.1.3 - June 5, 2020

refactor: stop sherlock checking for transient database triggers if running in non-update mode
fixed: another sdss photoz issue where photoz ranking about specz

v2.1.2 - May 24, 2020

fixed: merged result parameters are now merged correctly

v2.1.1 - May 21, 2020

fixed: fixes to cl utils

v2.1.0 - May 20, 2020

Now compatible with Python 3.*