NAV Navbar
cURL Python
  • Introduction
  • API Overview
  • Authentication
  • Compression
  • Base Path
  • API End Points
  • front page

    Introduction

    Welcome to GovScan.Info!!

    It's a project I started as a hobby to audit the use of TLS across .gov.my domains. The project now scans more than 4000 sites daily, and stores the details of each scan including

    All data from all scans exposed via the API. Users can query historical information (up to Apr-2018) till the latest scan results. Scans run from 01:00 to roughly 10:00 (GMT +8), after which they're made available for download via files, queried via API or just searched from the website

    API Overview

    You can use our API to access current and historical information of Gov TLS Audit.

    We have reference code in Python and Shell.You can view code examples in the dark area to the right, and you can switch the programming language of the examples with the tabs in the top right.

    This example API documentation page was created with Slate. Feel free to edit it and use it as a base for your own API's documentation.

    Authentication

    The API is fully public and requires no authentication. It does not validate user-agents, http headers or ip addresses, everything is accepted.

    Compression

    The API supports the following content codings:

    It also supports the following Accept-Encoding header format, according to the RFC 7231 specification:

    Base Path

    The Base URL for the API is located at https://govscan.info/api/v2/

    The URL can only be accessed via the https endpoint over port 443. Any attempt to query the API over http will result in a 301 redirect to https with no content in the response.

    Currently the API supports TLS1.0 or higher.

    TLS1.0 will be deprecated in 2018, please use TLS1.2 (or higher) instead. The API does not support SSLv3 (or lower).

    API End Points

    GET /listScans

    import requests
    
    response = requests.get('https://govscan.info/api/v2/listScans')
    print(response.text)
    
    curl "https://govscan.info/api/v2/listScans"
    
    

    JSON Response:

    {
      "files": [
        {
          "fileName": "scan_2018-06-03.zip",
          "url": "https://gov-tls-audit.sayakenahack.com/files/scan_2018-06-03.zip",
          "fileSize": "2.54 MB"
        },
        {
          "fileName": "scan_2018-06-02.zip",
          "url": "https://gov-tls-audit.sayakenahack.com/files/scan_2018-06-02.zip",
          "fileSize": "2.55 MB"
        }
      ]
    
    }
    
    

    Overview

    List of all scan files available for download.

    Scan files are produced daily, and have the full data of all sites scanned on a particular day. The files are available in csv, json and jsonl formats -- and all are available via a single zip file (per day).

    The API doesn't return the scan files, but the names and locations of all scan files.

    Request Parameters

    None

    Response Parameters

    Response is single json object names files, that is an array sorted by date in descending order, i.e. the latest file is the first element of the array.

    Each element of the array is a json object that has the following properties:

    Property Type Description Optionality
    fileName string The name of the zip file. The zip contains all 3 formats (csv, json, jsonl) Always Present
    url string Full url to retrieve the zip file from. URL is already fully formed, and can be used directly with wget Always Present
    fileSize string Size of the file Always Present

    GET /listFQDNs

    import requests
    
    response = requests.get('https://govscan.info/api/v2/listFQDNs')
    print(response.text)
    
    curl "https://govscan.info/api/v2/listFQDNs"
    
    

    JSON Response:

    {
      "FQDNS": [
        "1gfmasmigration.anm.gov.my",
        "1govserv.1govnet.gov.my",
        "1malaysiamap.mygeoportal.gov.my",
        "1mdb.islam.gov.my",
        "youth.bernama.com",
        "zakat.sabah.gov.my",
        "zakat4pengajian.muip.gov.my",
        "zimbra.maiwp.gov.my",
        "zimbra.prison.gov.my",
        "zmproxy1.treasury.gov.my"
      ]
    }
    

    Overview

    List all Fully Qualified Domain Names (FQDNs) that are scanned. The response is a flat array of FQDNs that are currently scanned daily. The response will not include older domains that have been removed, only domains that are currently being scanned daily. Hence it possible for FQDNs not in this list to exist in the API, they just won't have recent entries.

    Full list of FQDNs can also be viewed here

    Request Parameters

    None

    Response Parameters

    Response is a flat array of FQDNs

    Parameter Type Description Optionality
    Fully Qualified Domain Name String Fully Qualified Domain Name Always Present (Array)

    GET /listDNs

    import requests
    
    response = requests.get('https://govscan.info/api/v2/listDNs')
    print(response.text)
    
    curl "https://govscan.info/api/v2/listDNs"
    
    

    JSON Response:

    {"count": 613, "DNs": [
        "melakasafefood.gov.my", 
        "ptgwp.gov.my", 
        "politeknik.gov.my", 
        "bhgjjkrpahang.gov.my", 
        "pknk.gov.my", 
        "mbpj.gov.my", 
        "hebpenang.gov.my", 
        "mdi.gov.my", 
        "btpnns.gov.my", 
        "bog.gov.my", 
        "mastic.gov.my", 
        "pknp.gov.my"]
    }
    

    Overview

    List all Domain Names (FQDNs) that are scanned. The response is a flat array of DNs that are currently scanned daily. The response will not include older domains that have been removed, only domains that are currently part of the daily scan. Hence it possible for DNs not in this list to exist in the API, they just won't have recent entries.

    Full list of FQDNs can also be viewed here

    Request Parameters

    None

    Response Parameters

    Response is a flat array of FQDNs

    Parameter Type Description Optionality
    DNs String Domain Names in a flat array Always Present (Array)
    count Int Count of Domain Names present

    GET /siteLatest

    import requests
    
    response = requests.get('https://govscan.info/api/v2/siteLatest?FQDN=pengundi.spr.gov.my')
    print(response.text)
    
    curl "https://govscan.info/api/v2/siteLatest?FQDN=pengundi.spr.gov.my"
    
    

    JSON Response:

    Single Scan Object

    Overview

    Get the latest scan for the particular site. A site gets scanned everyday, and all details of every scan is available via the API. This endpoint provides the latest available scan in the database for a particular site.

    Request Parameters

    Parameter Type Description Optionality
    FQDN String Fully Qualified Domain Name Mandatory

    Response Parameters

    Single Scan Object

    GET /siteHistory

    import requests
    payload = {'FQDN':'pengundi.spr.gov.my', 
               'scanDate': '2018-04-15T01:20:01.686065',
               'limit': 5}
    response = request.get('https://govscan.info/api/v2/siteHistory', params = payload)
    print(response.text)
    
    curl "https://govscan.info/api/v2/siteHistory?
    FQDN=pengundi.spr.gov.my&
    scanDate=2018-04-15T01:20:01.686065&limit=5"
    
    

    JSON Response:

    {
      "results": [
        {
          "scanDate": "2018-04-14",
          "scanDateTime": "2018-04-14T11:58:28.669263",
          "TLSRedirect": true,
          "TLSSiteExist": true,
          "htmlSize": "7673",
          "siteTitle": "SURUHANJAYA PILIHAN RAYA MALAYSIA (SPR) - SEMAKAN DAFTAR PEMILIH"
        },
        {
          "scanDate": "2018-04-13",
          "scanDateTime": "2018-04-13T22:27:57.491373",
          "TLSRedirect": true,
          "TLSSiteExist": true,
          "htmlSize": "7671",
          "siteTitle": "SURUHANJAYA PILIHAN RAYA MALAYSIA (SPR) - SEMAKAN DAFTAR PEMILIH"
        },
        {
          "scanDate": "2018-04-13",
          "scanDateTime": "2018-04-13T03:37:46.727488",
          "TLSRedirect": false,
          "TLSSiteExist": false,
          "htmlSize": "4024",
          "siteTitle": "SURUHANJAYA PILIHAN RAYA MALAYSIA (SPR) - SEMAKAN DAFTAR PEMILIH"
        },
        {
          "scanDate": "2018-04-13",
          "scanDateTime": "2018-04-13T01:20:01.686065",
          "TLSRedirect": false,
          "TLSSiteExist": false,
          "htmlSize": "4024",
          "siteTitle": "SURUHANJAYA PILIHAN RAYA MALAYSIA (SPR) - SEMAKAN DAFTAR PEMILIH"
        },
        {
          "scanDate": "2018-04-12",
          "scanDateTime": "2018-04-12T01:20:01.612568",
          "TLSRedirect": true,
          "TLSSiteExist": true,
          "htmlSize": "10888",
          "siteTitle": "7.  SEMAKAN PEMILIH ISI RUMAH"
        }
      ]
    }
    

    Overview

    Get the history (summary only) of daily scans on a site.

    Request Parameters

    Parameter Type Description Optionality
    FQDN String Fully Qualified Domain Name Mandatory
    scanDate dateTime The starting dateTime to retrieve records from. API will return all scans older than this dateTime. If scanDate is not provided, the default is set to now. Optional
    limit integer The Maximum number of records to return. If limit is not provided, the default is 30 Optional

    Response Parameters

    Parameter Type Description Optionality
    results JSON Object results from query Always Present
    results. scanDateTime dateTime Date of scan in ISO 8601 Notation Always Present
    results. scanDate date Date of scan in yyyy-mm-dd format Always Present
    results. TLSRedirect boolean Indicator of whether site redirects from http to https Always Present
    results. TLSSiteExist boolean Indicator of whether a TLS version of the FQDN exist Always Present
    results. htmlSize integer Size (in Bytes) of the entire HTML (text only) on the root page Always Present
    results. siteTitle string Title of site (based on html title tages) Optional

    GET /scanDetails

    import requests
    payload = {'FQDN':'pengundi.spr.gov.my', 
               'scanDate': '2018-04-15T01:20:01.686065'}
    response = request.get('https://govscan.info/api/v2/scanDetails', params = payload)
    print(response.text)
    
    curl "https://govscan.info/api/v2/scanDetails?
    FQDN=pengundi.spr.gov.my&
    scanDate=2018-04-15T01:20:01.686065"
    
    

    JSON Response:

    Single Scan Object

    Overview

    Get all details of a specific scan, full details of a site at a particular point in time.

    Parameter Type Description Optionality
    FQDN String Fully Qualified Domain Name Mandatory
    scanDate dateTime Full date time in ISO 8601 format Mandatory

    to obtain scanDate of a scan, use the /siteHistory function to the full list of scans for a particular FQDN.

    GET /DNSHistory

    import requests
    
    response = requests.get('https://govscan.info/api/v2/DNSHistory?DN=example.gov.my')
    print(response.text)
    
    curl "https://govscan.info/api/v2/DNSHistory?DN=example.gov.my"
    
    

    JSON Response:

    {
      "results": [
        {
          "records": {
            "SOA": [
              "skmmns1.cmc.gov.my. support.cmc.gov.my. 2018012086 1200 600 1209600 3600"
            ],
            "TXT": [
              "\"v=spf1 a mx +all\""
            ],
            "NS": [
              "skmmns1.cmc.gov.my.",
              "skmmns2.cmc.gov.my."
            ]
          },
          "scanRecordTypes": ["SOA", "TXT", "MX", "NS", "DNSKEY"],
          "scanDate": "2018-09-30T21:07:21.687385",
          "domain": "skmm.gov.my"
        },
        {
          "records": {
            "SOA": [
              "skmmns1.cmc.gov.my. support.cmc.gov.my. 2018012086 1200 600 1209600 3600"
            ],
            "TXT": [
              "\"v=spf1 a mx +all\""
            ],
            "NS": [
              "skmmns2.cmc.gov.my.",
              "skmmns1.cmc.gov.my."
            ]
          },
          "scanRecordTypes": ["SOA", "TXT", "MX", "NS", "DNSKEY"],
          "scanDate": "2018-09-30T09:06:47.280562",
          "domain": "skmm.gov.my"
        }
      ]
    }
    

    Overview

    Returns the DNS History of a specific Domain Name (DN). All domains are scanned beginning 9.05pm MYT daily, and each record is stored in a historical table.

    Request Parameters

    Parameter Type Description Optionality
    DN String Domain Name Mandatory
    scanDate dateTime The starting dateTime to retrieve records from. API will return all scans older than this dateTime. If scanDate is not provided, the default is set to now. Optional
    limit integer The Maximum number of records to return. If limit is not provided, the default is 30 Optional

    Response Parameters

    Response is a flat array of FQDNs

    Parameter Type Description Optionality
    results Array Array of JSON object Always Present
    results[].scanRecordTypes Array Array of record types queried in this scan Always Present
    results[].scanDate datetime DateTime in ISO Format of the scan time Always Present
    results[].domain string Domain of the scan, will always be equal to DN Requested Always Present
    results[].records JSON Object Object to hold DNS Records
    results[].records.xx Array Array holding all entries corresponding to DNS records of type xx Always Present

    Scan Object

    JSON Response:

    {
      "FQDN": "pengundi.spr.gov.my",
      "scanDate": "2018-08-05T04:18:41.026247",
      "dateAdded": "2018-08-05",
      "TLSSiteExist": true,
      "TLSRedirect": true,
      "redirectType": "HTTP Status Code",
      "ip": "103.8.163.104",
      "domain": "spr",
      "siteTitle": "SURUHANJAYA PILIHAN RAYA MALAYSIA (SPR) - SEMAKAN DAFTAR PEMILIH",
      "formFields": "|ic|captcha|token",
      "links": [
        "https://play.google.com/store/apps/details?id=com.mysprsemak.spr",
        "https://itunes.apple.com/my/app/myspr-semak/id1262263101?l=ms&mt=8"
      ],
    
      "httpRequest": {
        "requestTime": "2018-08-05T04:18:41.026512",
        "headers": {
          "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
        },
        "requestUrl": "http://pengundi.spr.gov.my"
      },
    
      "httpResponse": {
        "htmlHash": "02d4624a296bd999789072f7a0f48498bad04580",
        "headers": {
          "Transfer-Encoding": "chunked",
          "Keep-Alive": "timeout=5, max=500",
          "X-Frame-Options": "DENY",
          "Cache-Control": "no-store, no-cache, must-revalidate",
          "Server": "Apache",
          "Connection": "Keep-Alive",
          "Set-Cookie": "PHPSESSID=07212f35da4917fb478e2f4972ee8d41; path=/",
          "Expires": "Thu, 19 Nov 1981 08:52:00 GMT",
          "Pragma": "no-cache",
          "Date": "Sat, 04 Aug 2018 20:18:41 GMT",
          "X-Powered-By": "PHP/7.1.17",
          "Content-Type": "text/html; charset=UTF-8"
        },
        "htmlSize": "8157",
        "url": "https://pengundi.spr.gov.my/",
        "statusCode": "200"
      },
    
       "certData": {
        "notValidBefore": "2018-04-25T08:40:42",
        "serialNumber": "79:D8:5B:0E:25:38:37:78:00:00:00:00:50:E2:18:96",
        "subject": {
          "commonName": "*.spr.gov.my",
          "countryName": "MY",
          "localityName": "Putrajaya",
          "organizationName": "Suruhanjaya Pilihan Raya Malaysia"
        },
        "notValidAfter": "2020-04-25T09:10:42",
        "certMatchesHostname": true,
        "signatureHashAlgorithm": "sha256",
        "statusMessage": "Pass",
        "issuer": {
          "commonName": "Entrust Certification Authority - L1K",
          "countryName": "US",
          "organizationName": "Entrust, Inc.",
          "organizationalUnitName": "("
        },
        "statusCode": "1"
       },  
    
      "TLSServerInfo": {
        "TLSServerNameIndication": "pengundi.spr.gov.my",
        "hostname": "pengundi.spr.gov.my",
        "highestTLSVersionSupported": "5"
      },
    
      "shodan": {
        "ipStr": "103.8.163.104",
        "ports": [
          "443",
          "80"
        ],
        "asn": "AS17564",
        "asnCountryCode": "MY",
        "lastUpdate": "2018-07-25T01:51:10.666491",
        "isp": "GITN Sdn. Bhd."
      } 
    }
    

    The Scan Object is a JSON Object that represents all data collected from a single scan of a specific FQDN. The Scan Object is returned in the response for /siteLatest and /scanDetails.

    Parameter Type Description Optionality
    FQDN string Fully Qualified Domain Name Always Present
    scanDate datetime Date and Time of scan in ISO format Always Present
    dateAdded date Date that this record was added to the database Always Present
    TLSSiteExist boolean Indicator of whether a TLS version of the FQDN exist Always Present
    TLSRedirect boolean Indicator of whether site redirects visitors from http to https Always Present
    redirectType string If TLSRedirect is True , indicates what type of http redirect is present Optional
    ip string IP address of the DNS lookup on this FQDN Always Present
    domain string domain name of the site (without TLD suffix) Always Present
    siteTitle string Title of site (based on html title tages) Optional
    formFields string Name of all form fields on the root page of FQDN. Fields are ` ` delimited
    links array All links appearing on the root page of the FQDN. Optional
    httpRequest JSON Object JSON Object containing information of the HTTP Request Always Present
    httpRequest. requestTime datetime Date time of HTTP request in ISO Format Always Present
    httpRequest. headers JSON Object Headers used in the request, only User-Agent is always present Always Present
    httpRequest. requestUrl string Requested URL (might not equl the response url) Always Present
    httpsRequest JSON Object If HTTP Request is not redirected to HTTPs, an explicit HTTPs request is made, httpsRequest has the same properties as the httpRequest Always Present
    httpResponse JSON Object JSON Object containing information of the HTTP Response Always Present
    httpResponse. htmlHash string SHA1 Hash of the entire HTML on the root page Always Present
    httpResponse. htmlSize integer Size (in Bytes) of the entire HTML (text only) on the root page Always Present
    httpResponse. headers JSON Object All HTTP headers returned in the response Optional
    httpResponse. url string Final URL of the response, this is after all HTTP redirects. This neglects any Javascript redirects present in the HTML Always present
    httpResponse. statusCode integer HTTP Status Code of the response, typically 200 since all redirects are followed Always Present
    httpsResponse JSON Object If HTTP Request is not redirected to HTTPs, an explicit HTTPs request is made, HTTPs Response has the same properties as the HTTP Response, but will only be present if httpsRequest is also present Optional
    certData JSON Object JSON object of Certificate Data pulled from HTTPs site. Only present if site actually has a working HTTPs implementation. Cert data is captured even if certificate is invalid Optional
    certData. notValidBefore datetime Not Valid Before field of Certificate in ISO Format Optional
    certData. serialNumber string Certificate Serial Number. Only Scans past 08-May-2018 return a valid Serial Number Optional
    certData. subject JSON Object Object of Subject, includes commonName, localityName, countryName..etc Optional
    certData. certMatchesHostname boolean True if the Hostname on the cert matches the actual hostname of the site. False otherwise. Optional
    certData. notValidAfter datetime Not Valid After field of Certificate in ISO Format Optional
    certData. signatureHashAlgorithm string Signature Hash Algorithm of Certificate Optional
    certData. issuer JSON Object Issuer Details of certificate Optional
    certData. statusCode Integer Certificate Status Code, 0 if cert is valid for site. Else an error code is provided Optional
    TLSServerInfo JSON Object TLS info on a Server level (not site) Optional
    TLSServerInfo. hostname string Hostname of server Optional
    TLSServerInfo. TLSServerNameIndication string Server Name Indication of Server Optional
    TLSServerInfo. highestTLSVersionSupported integer Highest version of TLS Supported Optional
    shodan JSON Object raw JSON from Shodan search result of IP Address Optional