Evaluations

An evaluation object represents a collection of Synapse Entities that will be processed in a particular way. This could mean scoring Entries in a challenge or executing a processing pipeline.

Imports:

from synapseclient import Evaluation, Submission, SubmissionStatus

Evaluations can be retrieved by ID:

evaluation = syn.getEvaluation(1901877)

Like entities, evaluations are access controlled via ACLs. The synapseclient.Synapse.getPermissions() and synapseclient.Synapse.setPermissions() methods work for evaluations:

access = syn.getPermissions(evaluation, user_id)

The synapseclient.Synapse.submit() method returns a Submission object:

entity = syn.get(synapse_id)
submission = syn.submit(evaluation, entity, name='My Data', team='My Team')

The Submission object can then be used to check the status of the submission:

status = syn.getSubmissionStatus(submission)
The status of a submission may be:

  • INVALID the submitted entity is in the wrong format

  • SCORED in the context of a challenge or competition

  • OPEN indicating processing has not completed

  • CLOSED indicating processing has completed

Submission status objects can be updated, usually by changing the status and score fields, and stored back to Synapse using synapseclient.Synapse.store():

status.score = 0.99
status.status = 'SCORED'
status = syn.store(status)

See:

Evaluation

class synapseclient.evaluation.Evaluation(**kwargs)

An Evaluation Submission queue, allowing submissions, retrieval and scoring.

Parameters

  • name – Name of the evaluation

  • description – A short description of the evaluation

  • contentSource – Synapse Project associated with the evaluation

  • submissionReceiptMessage – Message to display to users upon submission

  • submissionInstructionsMessage – Message to display to users detailing acceptable formatting for submissions.

To create an Evaluation and store it in Synapse:

evaluation = syn.store(Evaluation(
    name="Q1 Final",
    description="Predict progression of MMSE scores for final scoring",
    contentSource="syn2290704"))

The contentSource field links the evaluation to its synapseclient.entity.Project. (Or, really, any synapse ID, but sticking to projects is a good idea.)

Evaluations can be retrieved from Synapse by ID:

evaluation = syn.getEvaluation(1901877)

…by the Synapse ID of the content source (associated entity):

evaluation = syn.getEvaluationByContentSource('syn12345')

…or by the name of the evaluation:

evaluation = syn.getEvaluationByName('Foo Challenge Question 1')
__init__(**kwargs)

Initialize self. See help(type(self)) for accurate signature.

Submission

class synapseclient.evaluation.Submission(**kwargs)

Builds an Synapse submission object.

Parameters

  • name – Name of submission

  • entityId – Synapse ID of the Entity to submit

  • evaluationId – ID of the Evaluation to which the Entity is to be submitted

  • versionNumber – Version number of the submitted Entity

  • submitterAlias – A pseudonym or team name for a challenge entry

__init__(**kwargs)

Initialize self. See help(type(self)) for accurate signature.

Submission Status

class synapseclient.evaluation.SubmissionStatus(id, etag, **kwargs)

Builds an Synapse submission status object. https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatus.html

Parameters
__init__(id, etag, **kwargs)

Initialize self. See help(type(self)) for accurate signature.