Getting started with setup

[devibri]

Sorry, wasn't sure if this belonged in discussion or issues since it might just be a personal setup problem -- happy to move it if needed!

Was looking to get a local build set up for contributing to the project, but am unfortunately having some trouble going through the steps in the README --

Dev environment: Windows 10, using VSCode and VSCode's built in Powershell terminal

• I've cloned the website repo https://github.com/govdirectory/website
• I've installed snowman and put the .exe in the same directory as the repo
• I followed the steps to get a recent cache in the website directory (so setup looks like ...\GitHub\website\.snowman\cache\[FOLDERS])

The problem comes when I try to build the server:

When I run .\snowman build in the website directory (the .\ is to enable permissions to run this in the VSCode terminal) the process runs for a long time (20 mins?) and then errors out with the following:

Building project with 50 views.
Error: Failed to render page at site/statistics/index.html Error: template: statistics.html:28:37: executing "main" at <query "account-data.rq" .qid.String>: error calling query: Post "https://query.wikidata.org/sparql": read tcp 192.168.0.137:65240->208.80.154.224:443: wsarecv: An existing connection was forcibly closed by the remote host.

I'm not sure why this error is happening--perhaps an issue with the cache and making too many requests? When I check the cache though, it looks like there are items there:

.\snowman cache
There are 16004 cache items

Any help would be appreciated. Thanks so much!

[Ainali]

Thanks for asking, here is a good place to ask.
It was likely just the Wikidata Query server that was having a bad moment yesterday. I just tried to build now and it worked. If you try again now, does it work for you?

[Abbe98]

It could be a Snowman on Windows thing, I know that somethings were broken for quite a while before a Windows user stepped up and fixed things. I will try to reproduce it once I get my hand on a Windows setup.

[devibri]

Thanks for the replies! I've run .\snowman build again just now, and gotten another error:

Building project with 50 views.
Received bad(HTTP: 500 Internal Server Error) response from SPARQL endpoint:
SPARQL-QUERY: queryStr=SELECT
  # note that SAMPLE in Blazegraph always picks the first object
  (SAMPLE(?website) AS ?website)
  (SAMPLE(?email) AS ?email)
  (SAMPLE(?phone) AS ?phone)
  (SAMPLE(?citizensInitiatives ) AS ?citizensInitiatives)       
  (SAMPLE(?foiUrl) AS ?foiUrl)
  (SAMPLE(?contactPage) AS ?contactPage)
  (SAMPLE(?regulatoryText ) AS ?regulatoryText)
  (SAMPLE(?regulatoryTextTitle ) AS ?regulatoryTextTitle)
  (SAMPLE(?personLabel) AS ?leadBy)
  (SAMPLE(?parentOrg) AS ?parent)
  ?wikipedia
WHERE {
  BIND(wd:Q12433 AS ?org)

  OPTIONAL { ?org wdt:P856 ?website }
  OPTIONAL { ?org wdt:P968 ?email }
  OPTIONAL { ?org wdt:P9732 ?citizensInitiatives }
  OPTIONAL { ?org wdt:P11266 ?contactPage }
  OPTIONAL { ?org wdt:P10214 ?foiUrl }

  OPTIONAL {
    ?org wdt:P92 ?regulatoryTextItem .
    ?regulatoryTextItem wdt:P953 ?regulatoryText ;
                        wdt:P1476 ?regulatoryTextTitle .
  }

  OPTIONAL { ?org wdt:P749 ?parentOrg }

  OPTIONAL {
    ?org wdt:P2388|wdt:P1313 ?office .
    ?positionStatement ps:P39 ?office .
    ?person p:P39 ?positionStatement .
    ?person wdt:P31 wd:Q5 .
    MINUS { ?positionStatement pq:P582 ?endtime }
    MINUS { ?person wdt:P570 [] }
    FILTER ( ?office != wd:Q30185 )
  }

  OPTIONAL {
    ?org wdt:P1329 ?phone .
    ?org p:P1329 ?phoneStatement .
        ?phoneStatement (ps:P1329) [] .
        FILTER(EXISTS { ?phoneStatement prov:wasDerivedFrom ?reference. })
  }

  OPTIONAL {
    ?wikipedia schema:about ?org ;
               schema:isPartOf <https://en.wikipedia.org/> .
  }

  SERVICE wikibase:label {
    # this might need to be updated when new countries are added
    bd:serviceParam wikibase:language "en,sv,mul" .
    ?person rdfs:label ?personLabel .
  }
} GROUP BY ?wikipedia

java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:292)
        at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:678)
        at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:275)
        at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:269)
        at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:195)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
        at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.wikidata.query.rdf.blazegraph.throttling.SystemOverloadFilter.doFilter(SystemOverloadFilter.java:84)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:50)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.wikidata.query.rdf.blazegraph.filters.QueryEventSenderFilter.doFilter(QueryEventSenderFilter.java:125)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:43)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.wikidata.query.rdf.blazegraph.filters.JWTIdentityFilter.doFilter(JWTIdentityFilter.java:66)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.wikidata.query.rdf.blazegraph.filters.RealAgentFilter.doFilter(RealAgentFilter.java:33)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.wikidata.query.rdf.blazegraph.filters.RequestConcurrencyFilter.doFilter(RequestConcurrencyFilter.java:50)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:503)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:750)

Error: Failed to render page at site/statistics/index.html Error: template: statistics.html:37:35: executing "main" at <query "organization-optional.rq" .qid.String>: error calling query: Received bad response from SPARQL endpoint

So again looks like it might be a problem fetching the data? When running an older version of snowman (5 or 6.0), it fails more quickly with the error:

Building project with 50 views.
Error: Failed to render page at site/index.html Error: template: landing.html:70:26: executing "main" at <query "countries.rq">: error calling query: The given query could not be found. countries.rq

But I'm not sure if the versions are meant to be backwards-compatible. EDIT: Or it could be that the Windows fix you mentioned was introduced in 6.1?

[Abbe98]

Or it could be that the Windows fix you mentioned was introduced in 6.1?

This was indeed the case! https://github.com/glaciers-in-archives/snowman/releases/tag/0.6.1

[Abbe98]

Received bad(HTTP: 500 Internal Server Error) response from SPARQL endpoint:

This is actually a great sign! This means that everything works but query.wikidata.org/sparql is having trouble. This has gotten rather common I'm afraid, the solution however is to just run the build command again until the build is able to finish. Because Snowman will cache all the successful responses it will eventually complete but for the first build it might take a while.

[devibri]

Thanks! Out of curiosity, set up my system to dual boot into Linux and tried the commands again on there, which worked! Going to close this out as resolved.