…nio#4612)

Just some fixed MAX_BATCH_SIZE until we come up with something better.

Fixes: joernio#4611

…#4615)

…ariables (joernio#4617)

* Add reproducing test

* Add tests for new lambdas

* Never add Unknown node for captured variable

* Add todos to comments about lambda parameter capturing

* [python] prevent overtainting dictLiterals

* use allAssignmentTypes in isAssignment

* use methodFullNameExact instead of nameExact in isAssignment

As pointed out in joernio#4240, combining this nested immutable map-like structure has a quadratic performance, and the more performant strategy would be to use nested data-structures to merge.

For now, I've decided not to opt for a builder pattern, but rather keep the underlying structure mutable, and accessor methods return immutable structures.

)

This PR does two things:
1) DEBUG logs the frontend name, version and raw arguments passed down
2) INFO logs the CPG output path if not set explicitly by the user

Added a recursive aspect to the call type recovery if the receiver of the call is possibly recoverable (and it's receiver if that's a call and so on)

- log cpg creation for each file in DEBUG only instead of INFO
 - log the astgen path only once with INFO
 - removed logging that astgen runs now in the given input folder

…io#4628)

- Fix last error token position if error token itself is semicolon,
  newline or EOF.
- Also log the parser exception message.

…gitlab (joernio#4631)

Moved parser logging of memory clearing from INFO to DEBUG

A common pattern in JavaScript is something like

```javascript
var foo = {};
foo.bar = {};
foo.bar.someFunc = function someFunc() {};

foo.bar.someFunc();
```

This PR adds a post-processing pass to find instances where the definition and the call live in the same file and to link them. This "common file" limitation aims to reduce false linking.

also to minify the diff for the flatgraph port

Allow the frontend to continue analysis if the type stubs for built-ins is missing.

The `new` operator can prefix an ordinary call, e.g.
```javascript
var Print = function(str) {
	console.log(str)
}

new Print("Hello")
```
The type recovery pass only considers object instantiations from type declarations on `new` calls, but this change allows it to consider functions too.

Other changes include adding constants for `:program` and `<operator>.new`

Added implicit self receiver to simple calls.

In ruby as for all other dynamic language there is no vtable and thus
only a single entry in the binding tables with name=="" and
signature=="".

This change moves the type/method identifier references for entities exportable from the script to prefix the respective entity at the definition.

this caused me some headaches as I had an old jar lying around there,
likely from a bisect session...

…#4643)

This change moves the type/method identifier references for entities exportable from the script to prefix the respective entity at the definition.

ICPPASTExpression.getEvaluation may fail throwing an unrecoverable exception in case of unresolved includes etc.

* Added `Member` nodes for each method, to relate to their respective bound `TypeDecl` nodes.
* Added a type decl for "fake methods" that include the methods and classes exported by the file

To simplify the call graph, methods resolved to the `Kernel` class are dispatched as static.

Additionally, replaced `__builtin` with `Kernel` to map more closely to Ruby's internals and implemented `MemberAccessCommandContext`.

…ents (joernio#4649)

…oernio#4651)

* [javasrc2cpg] add AstCreator `code` implementation without comments

* add custom PrinterConfiguration to be used by `code`

…o#4654)

* [ruby] Removed catch-all check in namespaceToTypeMap, some tests will have to fixed with different passes

* [ruby] add failing test to be fixed at a later date

* [ruby] Use File.separator in RubyScope instead of hardcoding / char

* [ruby] Resolve windows paths for ruby imports

…ype (joernio#4656)

The starting point here was a discrepancy wrt `methodFullName`s in Python for the following two samples:

```python
a = 10
foo(a) # methodFullName = __builtin.int.foo
```
vs
```python
foo(10) # methodFullName = <unknownFullName>
```

It was noticed that in `setTypeInformationForRecCall` the argument `i` isn't always set according to the convention that the receiver should have argument index 0 when it's a DYNAMIC_DISPATCH call and 1 when it's a STATIC_DISPATCH. This patch requires the convention to be satisfied. In turn, had to ignore 2 (deprecated) Ruby tests that were not verifying this convention.

1. Created classes and modules in pairs, where one is `Foo` (the "regular" class) and the other is `Foo<class>` (the singleton class). `module` singletons get the `final` keyword.
2. Define members for `@@` fields under `Foo<class>`
3. Define members with `dynamicTypeHintFullName`s set to the corresponding `self` methods and under `Foo<class>`. `Foo<class>` should have empty bindings to these `self` methods. An empty binding is one with `name==""` and `signature==""`
4. During object instantiations, e.g. `Foo.new`, add `Foo<class>` to the receiver's dynamic type hints, and `Foo` as the return type of `Call(new)`. The call linker may add an edge directly from `Foo.new` to `Foo.initialize`.

cc @AndreiDreyer 

Resolves joernio#4652

* [ruby] import and dependency file node creator made

* [ruby] renamed linker pass to file pass

* [ruby] Added base file for dependency summary solver pass

* [ruby] first iteration for adding types and method nodes from ruby dependency summary

* [ruby] Finished dependency solver pass, added test cases. Removed manual file creation pass for imports, handled by FileCreationPass base pass

* [ruby] External dependencies naming done, import resolver is picking up most of the methods.

* [ruby] Moved import resolving tagger test to sendgrid until logger is fixed

* [ruby] Fixed failing tests

* [ruby] External dependency renaming finished, one importing test failing that needs to be fixed

* [ruby] Added stubbed type check for adding type in scope, program summary seems to be losing RUbyStubbedType though

* [ruby] added check for external types when deciding to use fullName, or build fullName. Fixed failing import test

* scalfmt

* [ruby] Changed / to JFile.separator

joernio#4657)

fixes joernio#4661

…lException (joernio#4665)

* Wrap all javaparser getType calls in Try

* Log failures

* Switch missed try

* Limit logging counts

* Fix one more crash and limit log count

* Use atomic operation for updating counts

* fix jssrc astgen invocation on linux/aarch64

I don't like the amount of code we have there that pretends as if "ARM"
is a single architecture... so I've moved things in a slightly better
direction renaming some constants to ARMv8. But I'm not going to fix
that all in one day.

* scalafmt

* add platforms to file name variables

* `initialize` methods are under all types/modules with fields that need to be initialized, where `InstanceFields` under classes are considered `ClassFields` and places under the singleton type. e.g, `class Foo;SomeMember=1;end` is considered under the singleton.
* `MemberAccess` is now not always treated as a call. As per Ruby, if the first letter of the member is capitalized, then it is a field access, otherwise a function call.
* Calls (and qualified calls) have their base/targets recursively checked for if they need to be prepended with `self.` (i.e., not a local variable that has been declared in scope).
* Added initial `<body>` method to catch all arbitrary statements under a type declaration, requires follow-up work
* Fixed local variable scoping, where variables introduced in control structure blocks get fixed to the parent method and not the block itself.

As discussed in e.g. joernio#4625 (comment)
we want to avoid having classpath dependencies on the frontends and instead invoke them
frontends as external processes (i.e. execute their start script). Otherwise we'll end in
jar hell with various incompatible versions of many different dependencies, and complex
issues with things like OSGI and JPMS.

There's no perfect way to do so, this is the one I currently like the most...

Fixes tab completion bug joernio#4625

…rnio#4668)

* Partial gosrc2cpg frontend perforamnce optimisations

1. Changed parsing of AST json inside AST Creator itself.
2. Made changes in download dependency processing to generate AST only
for the used packages and subsequently only processing used packages.
There was a change required inside `goastgen` utility to support the
`-include` input option.

TODO:
At this moment `AstCreator` gets instantiated while first-level
processing for building the cache and it stays in memory till
`AstCreationPass` is done. In subsequent changes, we will split the
processing in such a way that we create `AstCreator` for that pass and
destroy it once it's used. Instantiate it again for `AstCreationPass`.

The above change is done as part of the larger change.

* small pending change

* Refactored some code to use meaningful name in the context

…calls in AstCreator (joernio#4677)

This PR handles:
 * Moved values for `typeFullName` to `possibleTypes` during the `AstCreator` pass
 * Add handling for special naming convention of builtin methods in `RubyTypeHintCallLinker` pass

---------

Co-authored-by: David Baker Effendi <dave@whirlylabs.com>

…4680)

* [dataflowengineoss] refine return value semantics honoring

* fix c2cpg test

* make sure parentNode is an argument of childNode

Benchmarks have been moved to a standalone tool at https://github.com/joernio/joern-benchmarks

* Method-Members for singletons are now implemented and bound to their respective types.
* Nested type decl members are under singletons now
* TypeDecl members hint towards singletons now
* Removed `:program` Type Decl

* reduce classpath dependencies from frontends: jssrc2cpg

similar to joernio#4672, with one
additional refactor: reuse the scopt parser rather than manually
fiddling with the commandline args

* refactor

* reuse scopt parser for javasrc as well

* inline ParameterNames.TypePropagationIterations etc. again

* fixup

* add back old api as @deprecated to help users migrate

This PR handles:
 * Create a new `<body>` `MethodDecl` in `RubyNodeCreator` that holds all statements that aren't `methods`/`singleton methods`/`type decls` which was previously put under the `initialize` and `initialize<class>` methods manually
 * Invoke a call to `self::className::<body>` after the `TypeDecl` is created

Resolves joernio#4683

…ernio#4684)

* php

* pysrc

* swiftsrc

* symlink for php passes

* add previously missing swiftsrc2cpg part

* move known_function_signatures.txt to x2cpg resources

* php tests: bring back old behaviour, i.e. default iterations=3

Moved nested methods and type decls to be connected directly to the surrounding method via `AstLinker` instead of against the `Block` of the method.

* [python] type-recovery for index accesses of calls

* scalafmt

* [c2cpg] Improve full names and signatures for C++.

The C++ method full names did not include signatures which caused a lot
of methods to have the same full name which is not allowed. The
situation is not yet fully resolved because method/function declarations
independent from the implementation still created method stubs with full
name collisions. But it is a step in the right direction and the stubs
are easy to filter out via `isStub` step.

The call node generation has been reworked to accomodate this change.
Most noteable changes there:
- Full names where required contain signatures
- Signatures do not contain method names anymore
- Dispatch type is properly set if known.

* Address PR comments.

* Go memory usage optimisations (#31)

* Partial gosrc2cpg frontend perforamnce optimisations

1. Changed parsing of AST json inside AST Creator itself.
2. Made changes in download dependency processing to generate AST only
for the used packages and subsequently only processing used packages.
There was a change required inside `goastgen` utility to support the
`-include` input option.

TODO:
At this moment `AstCreator` gets instantiated while first-level
processing for building the cache and it stays in memory till
`AstCreationPass` is done. In subsequent changes, we will split the
processing in such a way that we create `AstCreator` for that pass and
destroy it once it's used. Instantiate it again for `AstCreationPass`.

The above change is done as part of the larger change.

* small pending change

* Refactored some code to use meaningful name in the context

* partial changes

* yet to fix downloaddependencytest issue

* Fix for the final failing unit tests

* Fixes for review comments

* review coment fixes

…lelCpgPass (joernio#4689)

…rnio#4698)

* Builtins are renamed from `__builtin` -> `__core` to more closely resemble the core gem in Ruby
* Removed surrounding `<` and `>` tags from builtin packages
* Handling `Array:[]` type calls as `Array.[]` calls

…4693)

* use scala primitives (e.g. Integer -> Int) and some cleanup

motivation: minify diff for flatgraph migration

* latest cpg

* latest cpg

* upgrade cpg

* released cpg

…o#4699)

* Cpg.newDiffGraphBuilder: use new api to minify flatgraph diff

* import

Given the code block below:
```ruby
def foo &block
 puts block.call
end

foo do
 "world!"
end

Proc parameters from do-blocks (interpreted with the usual CPG lambda handling), suggests that there is a `call` member on the type bound to the type decl.

This PR adds this member, with the dynamic type hint referring to the type decl bound to this method.

```
Resolves joernio#4700

…nio#4703)

Earlier changes to include packages with include regex, didn't filter all the packages and had a limitation to include all sub folders/packages if root package is being used. However, that wasn't the case. Hence, we introduced a mechanism to check the entire package folder path with `goastgen` with a separate flag `-includePacakges` and updated integration.

* upgrade deps

* upgrade cpg

* upgrade and adapt

Merge 2-0-427 to backup_sync

They may fail throwing an unrecoverable exception in case of unresolved includes etc.
Also, some minor clean-up.

For: https://shiftleftinc.atlassian.net/jira/software/c/projects/SEN/issues/SEN-2777

For: https://shiftleftinc.atlassian.net/browse/SEN-2838

This PR handles:
 * Move parser tests from the `deprecated` frontend to the new `ruby` frontend.
 * Fixed parser issues with HashLiterals
 * Fixed parser issues with one-liner class definitions
 * Fixed parser issues with arguments in functions

Brings in latest babel and typescript to astgen.

* [ruby] Arrow Lambda Parameter Fix
Tests various lambdas and fixes `self` parameter name in methods.

partly to minify the flatgraph diff

* minor work on reachingDef

* fmt

---------

Co-authored-by: Michael Pollmeier <michael@michaelpollmeier.com>

This astgen version skipps giant, unparsable files with EMSCRIPTEN code now by default.

For: https://shiftleftinc.atlassian.net/browse/SEN-2797

…joernio#4719)

For: https://shiftleftinc.atlassian.net/browse/SEN-2777

Also handle fullnames with generics correctly when stubbing types
For: https://shiftleftinc.atlassian.net/browse/SEN-2840

This PR adds a few more parser tests based on tests in the `querying/` folder for Ruby.

Inheritance via `<` in Ruby can be arbitrary extensions which warrant post-processing analysis, so this removes any attempt to resolve the type at AST creation to allow for a post-processing pass to handle this instead.

Does not exist anymore.

As the `<body>` call is synthetic and meant to be immediately deterministic, so there is no reason it should be re-determined.

* upgrade cpg and adapt

* use released cpg

…joernio#4729)

```
[warn] -- [E092] Pattern Match Unchecked Warning:
/home/mp/Projects/shiftleft/joern.1/joern-cli/frontends/pysrc2cpg/src/test/scala/io/joern/pysrc2cpg/PySrc2CpgFixture.scala:74:61
[warn] 74 |    path.resultPairs().collect { case (firstElement: String,
secondElement: Option[Integer]) =>
[warn]
|                                                             ^
[warn]    |the type test for Option[Integer] cannot be checked at
runtime because its type arguments can't be determined from Option[Int]
[warn]    |
[warn]    | longer explanation available when compiling with `-explain`
}
```