1. Simulation Curriculum Corp.
  2. Community
  3. Announcements
New post
1

GAIA DR-2 star data in SkySafari

Hi everyone.

Now that the second data release (DR-2) of the GAIA star catalog is available, we are working on incorporating it into SkySafari Pro.

The "standard" SkySafari Pro will include GAIA stars to G magnitude 15.  This is a modest increase in the number of stars (from 29 to about 33 million) that ship in SkySafari Pro by default.

The big whopper will come in the GAIA extension, which will replace the UCAC5 extension in-app purchase.  This will be a free replacement for the UCAC5 extension - if you already purchased the UCAC5 extension, you won't have to buy the GAIA DR2 extension separately.  The new GAIA DR-2 extension will include stars from G magnitude 15.0 to 17.0.  That's a total of about 120 million stars, as opposed to the ~80 million currently present in the UCAC5 extension.  The file size of the GAIA extension will be 2.24 gigabytes, as opposed to 1.45 gigabytes.

Beyond this, we are thinking we'll move to a "streaming" architecture, where even more of the GAIA catalog is loaded on demand, in bits and pieces, across the internet, as you pan around the sky.  The entire 1.7-billion-star GAIA catalog would occupy a file more than 30 gigabytes in size.  We just don't think that's practical, even in this day and age.  But we can certainly stream it as you need it.

We'd like to get your thoughts on whether we should just jump to a streaming architecture, right now, and don't bother with a 2.24GB downloadable "extension".  Is 2.24 GB simply too big of a file?  Do people really want this many stars downloaded to their mobile device?  Flame away!

Tim

Date Votes

25 comments

  • Avatar
    Tim DeBenedictis Official comment

    Thanks everyone.  Based on this feedback, here is what we're going to do in the near term.

    1) The "built-in" star data that comes with SkySafari Pro will contain GAIA stars down to Johnson V (visual) magnitude 15.0.  That's 25,162,273 GAIA stars.

    2) The "extended" star data associated with the in-app purchase will come as a single file download, 1.64 gigabytes in size.  It extends the star data to Johnson V magnitude 17.0, and adds another 89,871,826 stars (to the 25 million already baked into SkySafari Pro).

    3) In both GAIA star data files, we will convert the GAIA G, G_BP, and G_RP magnitudes to Johnson V magnitude.

    4) No proper motion, parallax, color, or radial velocity will be included at this time.  All GAIA star positions will be fixed at epoch J2000.  Adding the proper motions, etc. would nearly double the file sizes and (more critically) memory usage.  We'll consider adding those additional parameters in a future update if there's a strong demand for them.

    I really appreciate everyone's feedback here.  We're hoping to have this GAIA out within the next 2 weeks.

  • 2
    Avatar
    Chris

    I personally would much, much prefer the download to streaming. Anytime I really need stars beyond the brightest 25 million is when I'm at remote locations with little to no cell service. Additionally, I would prefer to download everything at home over Wi-Fi and not use my data plan unnecessarily. I like the idea of switching from UCAC to GAIA, as it's even more stars and presumably more accurate location and magnitude information? Switching to streaming is a move in the wrong direction in my opinion. I get that everything is going towards streaming as opposed to local storage, but we can't lose sight of this being an astronomy app used in very remote locations where Wi-Fi isn't available.

    File sizes are a problem for many, but not for me. Maybe if you could make it possible to store the additional databases on an external memory card that would help some people.

  • 2
    Avatar
    JoeTimmerman

    I completely agree with Chris.  No to streaming.  Yes to local downloads.

    Joe

  • 1
    Avatar
    Owen

    I would agree to the download option. I am not sure about streaming as many places don't have useful data connections. However and this is a big but I am not sure how useful the GAIA green magnitudes are. There are decent transforms to get them to a V mag and I would prefer that rather than some magnitude that may not be close to anything we can see.

     

    Owen

  • 0
    Avatar
    Owen

    Should have said this is the formula for getting V mags from GAIA data

     

    V := G + 0.01760 + 0.006860 * (BPg−RPg) + 0.1732 * (BPg−RPg)^2

     

    Owen

  • 1
    Avatar
    Bentley Ousley

    I would prefer download, also. (or at least give us the _option_ to download the 2.24 gb file. ) Streaming doesn't work for me in the field for a couple of reasons,  1) the cellular data isn't always reliable and  2) My cell phone allows only Hotspot WiFi (no Bluetooth tethering ) I'm currently using a WiFi-only Android tablet for SkySafari and guess where the WiFi is connected? ... that's right ... to the Digital Setting Circles device.

     

     

  • 1
    Avatar
    Chris

    Thanks for that update Tim. Can you comment about the differences and advantages we'll see by the switch from UCAC to the GAIA catalog? Will we see a more complete catalog to the magnitudes you mentioned above? Are the positions and numbers used for V magnitudes more accurate?

    Separately, do you know if any work is planned to update the object catalogs with more accurate data? For example, the NGC/IC catalogs have had a ton of work done on them since the originals by various others and I consistently see large errors in how V mags have been calculated for those.

  • 1
    Avatar
    Welliewhitney755

    Yes Please to 2.24GB downloadable "extension" and the rest for "streaming" architecture. 

    I would prefer to download 2.24GB downloadable "extension" so to have it at hand when I got no connection to Wi-Fi. 

    I am so happy that you are now going to use second data release (DR-2) of the GAIA star catalogue. This gives the best database of any astronomy app and worth every penny spent on it.

    I can't wait for the rest of the second data release (DR-2) of the GAIA star catalogue to be available for this app on Mac and Andriod.  

    Thanks for the hard work. 

     

  • 0
    Avatar
    Chris

    Hey Tim, was wondering if you could respond to my previous message above. Thanks 

  • 2
    Avatar
    Tim DeBenedictis

    Chris, sorry for the delayed response.  I've been recovering from a hard drive meltdown the past 24 hours.  Also, our app update which supports the GAIA catalog has been held up in app review for an unrelated reason.  In the meantime here are some answers:

    Q. Can you comment about the differences and advantages we'll see by the switch from UCAC to the GAIA catalog?

    A. Here are the old (UCAC5) vs. new (GAIA) comparisons:

    "The old"
     
    - UCACStars.skydat - the "bright" file, built into Pro, with stars down to U mag 15.0.  Total 26,270,186 stars, file size 473.2 MB.
    - UCACStars2.skydat - the "faint" file, in-app purchase, with stars fainter than U mag 15.0 (to about 16.5).  Total 80,403,781 stars, file size 1.45 GB.
    - "false double stars" caused by UCAC5 stars which duplicate stars in the brighter Hipparcos, Tycho-2, WDS, GCVS catalogs.  I had no good way to strip these since UCAC5 had no cross-reference to other catalogs.
    - Magnitudes are in the "U" (UCAC) system, which is close to (not quite the same as) the Johnson V (visual) system we use everywhere else.
    - No proper motion, parallax, color, or radial velocity included.  Positions fixed at J2000.
    - Positional accuracy test: Pluto Occultation 14 Aug 2018.  Pluto misses target star by 1.6 arcseconds.
     
    "The new"
     
    - GAIAStars.skydat - the "bright" file, built into Pro, with stars down to V mag 15.0.  Total 25,162,273  stars, file size 459.1 MB.
    - GAIAStars2.skydat - the "faint" file, in-app purchase, with stars from V mag 15.0 to 17.0.  Total 89,871,826 stars, file size 1.64 GB.
    - Stripped out all GAIA stars duplicated in the Hipparcos, Tycho, WDS, GCVS catalogs.  "false double stars" eliminated using GAIA <-> HIP <-> TYC2 cross-ref.
    - In both GAIA star data files, we've converted the GAIA G, G_BP, and G_RP magnitudes to Johnson V magnitude, for consistency
    - No proper motion, parallax, color, or radial velocity included.  Positions fixed at J2000.  If I add this stuff, it will essentially double the above file sizes.
    - Positional accuracy is significantly improved. Example Pluto occultation 14 Aug 2018; Pluto now misses by 0.1 arcseconds.

    Q. Separately, do you know if any work is planned to update the object catalogs with more accurate data? For example, the NGC/IC catalogs have had a ton of work done on them since the originals by various others and I consistently see large errors in how V mags have been calculated for those.

    A. Our source for NGC-IC data is Wolfgang Steinicke's Revised NGC-IC:

    http://www.klima-luft.de/steinicke/ngcic/ngcic_e.htm

    The most recent version of his data that we've pulled is from 2016.  I see he has a new version (Jan 2018) available.  I'll take a look at that sometime this week.

     

  • 0
    Avatar
    Chris

    Tim-- Thanks so much for providing this, this is very helpful.  I'm an Android user and seem to have already gotten the update, so I should be seeing the new GAIA catalog.  I had two follow-up questions that I'd like to better understand:

    1) When you say you stripped out all GAIA duplicate stars from the other catalogs you mentioned, are you leaving the GAIA information for those stars and stripping out the historical catalog information, or are you stripping out the GAIA and leaving the historical?  I was just curious if the GAIA info was the most accurate and up-to-date and which was being kept.  

    2) Have you found that the Johnson V mag to be more accurate than what we had in the UCAC catalog?  My guess here is yes, but I wasn't sure.

    I would very much like to see Wolfgang's most recent version implemented, if possible, as I'm consistently seeing objects that appear to be quite off on the V Mag shown when compared to his and others, like Steve Gottlieb's, calculated visual mags.  Thanks again!

  • 2
    Avatar
    Jeremiah Burton

    On an Android device a 30 GB download is no big deal, if it could be stored on an SD card.

  • 0
    Avatar
    Chris

    Hey Tim, any update on the two items above? Thanks

  • 0
    Avatar
    Tim DeBenedictis

    Chris,

    1) At the moment, we're using Hipparcos-2 data for everything in Hipparcos, Tycho-2 data for everything in Tycho-2, and GAIA-2 data for all the GAIA stars not in the Hipparcos/Tycho cats.  I think the GAIA data is generally considered to be more accurate than the Tycho-2 data, so we should probably replace all of the Tycho star data with GAIA star data at some point.  I think the accuracy of Hipparcos-2 is comparable to GAIA, and a lot of the very bright stars (i.e. naked eye stars) are missing from GAIA.  So I think we'll continue to use Hipparcos for the very brightest stars.

    2) I don't have any direct experience here.  Neither catalog contains "error bars" on the magnitude measurements.  I would, however, suspect the GAIA magnitudes to be better, since they're space-based measurements, not ground-based.  But I'm just guessing.

    Tim

  • 0
    Avatar
    Chris

    Thanks Tim, that's very helpful. I'll be watching the updates to see if you guys end up using Wolfgang's updated data.

  • 0
    Avatar
    Chris

    Hey Tim-- Wolfgang recently indicated that there will be a 2019 version coming out soon, possibly in the next few weeks.  Since it appears SkySafari wasn't updated to the 2018 version, is it possible we could get the database updated as soon as the new version comes out?  I did a quick, random check of some NGC's and noticed that a high percentage of them have changed between 2016 and 2018, and even more will change in the new version.  Thanks

    Chris

  • 0
    Avatar
    Bill Tschumy

    Chris,

    Is this post in the wrong spot?  What does this have to do with the GAIA data?

    Also note that Tim does not regularly read the Community Forums, but I will make sure he sees you message.

    Bill

  • 0
    Avatar
    Bill Tschumy

    Also note that this is a bigger task than just updating the data.  We  have 1000's of written descriptions and need to ensure they are changed as well.

  • 0
    Avatar
    Chris

    Hey Bill, thanks for your responses. I only posted here since the prior discussion evolved in this thread. As for the update, I'm mostly concerned with replacing the visual mags from your database for NGC/IC with his, since they're out of date and, often times, 1-2 mag off. I was previously choosing not to look for objects based on this number before realizing many objects are capable of being found by my equipment even if the SkySafari data suggests otherwise. I get that this was the best data reasonably available years ago, but it looks like Wolfgang has made vast improvements. I presume others are using the data as I am. Could just the vmags be updated until more time is available for a full update of descriptions, etc?

  • 2
    Avatar
    Bill Tschumy

    We are going to wait for Wolfgang to come out with his 2019 update and then incorporate it in SkySafari.  May be a couple of months.

  • 0
    Avatar
    Chris

    Hey Bill--

    Wolfgang has just recently posted his 2019 update.  Very much looking forward to seeing his data, especially his updated magnitudes and positions, incorporated into SS.   Appreciate you keeping us posted.

    http://www.klima-luft.de/steinicke/index_e.htm

  • 0
    Avatar
    Bill Tschumy

    Thanks for letting us know.  Still will be a couple of months out.  We are working on something else at the moment.  

  • 0
    Avatar
    Chris

    Sounds good. Is it a secret or can we have a hint as to what's coming?

  • 0
    Avatar
    Chris

    Hey Bill--

     

    I know you said it would be a couple of months on this, but I wanted to see if this would be in the update coming out, as I think getting accurate data in for NGC objects to be very important to have.  Thanks

    Chris

  • 0
    Avatar
    Bill Tschumy

    We have already incorporated Wolfgang's 2019 update in the database.  I'm planning on releasing this week if nothing else serious comes up.

Please sign in to leave a comment.