Revision control

Other Tools

1
# Introduction to WebKit
2
3
## What is WebKit?
4
5
[WebKit](https://webkit.org/) is an open-source Web browser engine.
6
It’s a framework in macOS and iOS, and used by many first party and third party applications including Safari, Mail, Notes, Books, News, and App Store.
7
8
The WebKit codebase is mostly written in C++ with bits of C and assembly, primarily in JavaScriptCore, and some Objective-C to integrate with Cocoa platforms.
9
10
It primarily consists of the following components, each inside its own directory in [Source](https://trac.webkit.org/browser/webkit/trunk/Source):
11
12
* **bmalloc** - WebKit’s malloc implementation as a bump pointer allocator. It provides an important security feature, called IsoHeap,
13
which segregates each type of object into its own page to prevent type confusion attacks upon use-after-free.
14
* **WTF** - Stands for Web Template Framework. WebKit’s template library.
15
The rest of the WebKit codebase is built using this template library in addition to, and often in place of, similar class templates in the C++ standard library.
16
It contains common container classes such as Vector, HashMap (unordered), HashSet, and smart pointer types such as Ref, RefPtr, and WeakPtr used throughout the rest of WebKit.
17
* **JavaScriptCore** - WebKit’s JavaScript engine; often abbreviated as JSC.
18
JSC parses JavaScript and generates byte code, which is then executed by one of the following four tiers.
19
Many tiers are needed to balance between compilation time and execution time.
20
Also see Phil's blog post about [Speculation in JavaScriptCore](https://webkit.org/blog/10308/speculation-in-javascriptcore/).
21
* **Interpreter** - This tier reads and executes instructions in byte code in C++.
22
* **Baseline JIT** - The first Just In Time compiler tier serves as the profiler as well as a significant speed up from the interpreter.
23
* **DFG JIT** - Data Flow Graph Just In Time compiler uses the data flow analysis to generate optimized machine code.
24
* **FTL JIT** - Faster than Light Just In Time compiler which uses [B3 backend](https://webkit.org/blog/5852/introducing-the-b3-jit-compiler/).
25
It’s the fastest tier of JSC.
26
JavaScriptCode also implements JavaScriptCore API for macOS and iOS applications.
27
* **WebCore** - The largest component of WebKit, this layer implements most of the Web APIs and their behaviors.
28
Most importantly, this component implements HTML, XML, and CSS parsers and implements HTML, SVG, and MathML elements as well as CSS.
29
It also implements [CSS JIT](https://webkit.org/blog/3271/webkit-css-selector-jit-compiler/), the only Just In Time compiler for CSS in existence.
30
It works with a few tree data structures:
31
* **Document Object Model** - This is the tree data structure we create from parsing HTML.
32
* **Render Tree** - This tree represents the visual representation of each element in DOM tree computed from CSS and also stores the geometric layout information of each element.
33
* **WebCore/PAL and WebCore/platform** - Whilst technically a part of WebCore, this is a platform abstraction layer for WebCore
34
so that the rest of WebCore code can remain platform independent / agnostic across all the platforms WebKit can run on: macOS, iOS, Windows, Linux, etc...
35
Historically, most of this code resided in WebCore/platform.
36
There is an ongoing multi-year project to slowly migrate code to PAL as we remove the reverse dependencies to WebCore.
37
* **WebKitLegacy** (a.k.a. WebKit1) - This layer interfaces WebCore with the rest of operating systems in single process and implements WebView on macOS and UIWebView on iOS.
38
* **WebKit** (a.k.a. WebKit2) - This layer implements the multi-process architecture of WebKit, and implements WKWebView on macOS and iOS.
39
WebKit’s multi-process architecture consists of the following processes:
40
* **UI process** - This is the application process. e.g. Safari and Mail
41
* **WebContent process** - This process loads & runs code loaded from websites.
42
Each tab in Safari typically has its own WebContent process.
43
This is important to keep each tab responsive and protect websites from one another.
44
* **Networking process** - This process is responsible for handling network requests as well as storage management.
45
All WebContent processes in a single session (default vs. private browsing) share a single networking session in the networking process.
46
* **WebInspector / WebDriver** - WebKit’s developer tool & automation tool for Web developers.
47
48
## Contributing to WebKit
49
50
There are many ways to get involved and contribute to the WebKit Project.
51
Filing a new bug, fixing a bug, or adding a new feature.
52
53
There are three different kinds of contributors in the WebKit project.
54
55
* Contributor - This category emcompasses everyone. Anyone who files a bug or contributes a code change or reviews a code change is considered as a contributor
56
* Committer - A committer is someone who has write access to [WebKit's subversion repository](https://svn.webkit.org/repository/webkit/).
57
* Reviewer - A reviewer is someone who has the right to review and approve code changes other contributors proposed.
58
59
See [Commit and Review Policy](https://webkit.org/commit-and-review-policy/) for more details on how to become a committer or a reviewer.
60
61
### Staying in Touch
62
63
Before getting in touch with WebKit developers using any of the avenues below, make sure that you have checked our page on how to ask [questions about WebKit](https://webkit.org/asking-questions/).
64
65
You can find WebKit developers, testers, and other interested parties on the [#WebKit Slack workspace](https://webkit.slack.com/).
67
and stay in touch.
68
69
## Bug tracking in WebKit
70
71
[bugs.webkit.org](https://bugs.webkit.org/) hosted is the primary bug tracking tool we use.
72
When making a code change, we post a code change (patch) on this website.
73
74
### Filing a bug and editing bugs
75
76
To [file a new WebKit bug](https://bugs.webkit.org/enter_bug.cgi), see [reporting bugs](https://webkit.org/reporting-bugs/).
77
78
To edit an existing bug, you may need [editbug-bits](https://webkit.org/bugzilla-bits/).
79
80
### Code Reviews in bugs.webkit.org
81
82
We also use [bugs.webkit.org](https://bugs.webkit.org/) to upload & review code changes to WebKit.
83
You can post a code change with `Tools/Scripts/webkit-patch upload`.
84
Note that the `webkit-patch` script only looks for changes below current directory,
85
so generally you should change the current directory to the top-level directory of a WebKit first.
86
87
When a patch is posted on [bugs.webkit.org](https://bugs.webkit.org/) requesting a formal code review (r? flag is set),
88
The Early Warning System (a.k.a. EWS) will automatically build and run tests against your code change.
89
This allows contributors to find build or test failures before committing code changes to the WebKit’s primary Subversion repository.
90
91
Once a patch is approved by a reviewer (r+ flag is set),
92
then the patch can be either committed directly into the Subversion repository by a WebKit committer,
93
who has write access to the Subversion repository,
94
or via the commit queue which can be requested by setting cq? flag and approved by any WebKit committer by setting cq+.
95
96
The Subvesion commit message should be created by `Tools/Scripts/commit-log-editor` based on the change log entries.
97
`Tools/Scripts/webkit-patch land` does this automatically.
98
99
### Security Bugs in bugs.webkit.org
100
101
Security bugs have their own components in [bugs.webkit.org](https://bugs.webkit.org/).
102
We’re also working on a new policy to delay publishing tests for security fixes until after the fixes have been widely deployed.
103
104
_***Do not post a patch or describe a security bug in a bug that is not in security component of bugs.webkit.org.***_
105
106
## Getting started with WebKit
107
108
### Getting Code
109
111
112
### Adding Tools to PATH
113
114
For convenience, you can add `Tools/Scripts/` to your path as follows in `~/.zshrc` like so:
115
116
```sh
117
export PATH=$PATH:/Volumes/Data/webkit/Tools/Scripts/
118
```
119
120
where `/Volumes/Data/webkit` is the path to a WebKit checkout.
121
122
This will allow you to run various tools you by name instead of typing the full path of the script.
123
124
### Updating checkouts
125
126
There is a script to update a WebKit checkout: `Tools/Scripts/update-webkit`. This script will automatically merge change logs mentioned below.
127
128
### Building WebKit
129
131
132
### Fixing mysterious build or runtime errors after Xcode upgrades
133
134
If you see mysterious build failures or if you’ve switched to a new version of
135
macOS or Xcode, delete the `WebKitBuild` directory.
136
`make clean` may not delete all the relevant files,
137
and building after doing that without deleting the `WebKitBuild` directory may result in mysterious build or dyld errors.
138
139
### Building with Address Sanitizer to investigate memory corruption bugs
140
141
To build [Address Sanitizer](https://en.wikipedia.org/wiki/AddressSanitizer) or ASan builds to analyze security bugs,
142
run `Tools/Scripts/set-webkit-configuration --asan --release`.
143
This will enable ASan build. If want to attach a debugger, you can also specify `--debug` instead of `--release`.
144
Once you don’t need to build or run ASan anymore, you can specify `--no-asan` in place of `--asan` to disable ASan.
145
Note that this configuration is saved by creating a file called Asan in the WebKitBuild directory,
146
so if you are trying to do a clean Asan build by deleting the build directory you need to rerun this command.
147
148
### Using Xcode
149
150
You can also use Xcode to build & debug WebKit. Open `WebKit.xcworkspace` at the top level directory.
151
152
In order to make Xcode use build files built by `make` command above,
153
go to File > Workspace Settings... > Advanced... > Custom > Relative to Workspace
154
and adjust the relative paths of Products and Intermediates to point to `WebKitBuild` directory.
155
![Screenshot of XCode Workspace Settings](resources/xcode-workspace-settings.png)
156
![Screenshot of XCode Workspace Settings - Advanced Build Location](resources/xcode-workspace-build-location.png)
157
Note that debugging WebCore code typically requires attaching to the relevant WebContent process,
158
not the application process, which is mostly running code in [Source/WebKit/UIProcess](https://trac.webkit.org/browser/webkit/trunk/Source/WebKit/UIProcess).
159
Depending on what you’re debugging, you’d have to attach & debug different processes in the coalition.
160
161
You may find it useful to use the debug helpers under `Tools/lldb/lldb_webkit.py`.
162
This can be added to `~/.lldbinit` for automatic loading into LLDB on launch by adding the line `command script import {Path to WebKit}/Tools/lldb/lldb_webkit.py`.
163
For more details, see the Wiki article on [lldb formatters](https://trac.webkit.org/wiki/lldb%20formatters).
164
165
When debugging a debug build in LLDB, there are also a few functions that can be called on objects that will dump debugging info.
166
167
* RenderObject
168
* showNodeTree()
169
* showLineTree()
170
* showRenderTree()
171
* Node
172
* showTree()
173
* showNodePath()
174
* showTreeForThis()
175
* showNodePathForThis()
176
177
## Correctness Testing in WebKit
178
179
WebKit is really big on test driven development, we have many types of tests.
180
181
* **JavaScript tests** - Resides in top-level [JSTests](https://trac.webkit.org/browser/webkit/trunk/JSTests) directory.
182
This is the primary method of testing JavaScriptCore. Use `Tools/Scripts/run-javascriptcore-tests` to run these tests.
183
* **Layout tests** - Resides in top-level [LayoutTests](https://trac.webkit.org/browser/webkit/trunk/LayoutTests) directory.
184
This is the primary method of testing WebCore.
185
If you’re making code changes to WebCore, you typically run these tests. Use `Tools/Scripts/run-webkit-tests` to run these.
186
Pass `-1` to run tests using WebKitLegacy (a.k.a. WebKit1).
187
[WebKitTestRunner](https://trac.webkit.org/browser/webkit/trunk/Tools/WebKitTestRunner) is used to run these tests for WebKit2,
188
and [DumpRenderTree](https://trac.webkit.org/browser/webkit/trunk/Tools/DumpRenderTree) is used to these tests for WebKit1.
189
There are a few styles of layout tests but all of them have a test file and expected result (ends with -expected.txt),
190
and the test passes if the test file’s output matches that of the expected result.
191
* **API tests** - Reside in [Tools/TestWebKitAPI](https://trac.webkit.org/browser/webkit/trunk/Tools/TestWebKitAPI),
192
these are [GTests](https://en.wikipedia.org/wiki/Google_Test) that test APIs exposed by JavaScriptCore,
193
WebKitLegacy, and WebKit layers as well as unit tests for selected WTF classes.
194
WebKit does not use [XCTests](https://developer.apple.com/documentation/xctest).
195
Use `Tools/Scripts/run-api-tests` to run these tests.
196
Because these API tests are sequentially, it’s preferable to write layout tests when possible.
197
* **Bindings tests** - Reside in [Source/WebCore/bindings/scripts/test](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/bindings/scripts/test),
198
these are tests for WebCore’s binding code generator.
199
Use `Tools/Scripts/run-bindings-tests` to run these tests.
200
* **webkitpy tests** - Tests for WebKit’s various Python scripts in [Tools/Scripts/webkitpy](https://trac.webkit.org/browser/webkit/trunk/Tools/Scripts/webkitpy).
201
Use `Tools/Scripts/test-webkitpy` to run these tests.
202
* **webkitperl tests** - Tests for WebKit’s various Perl scripts in [Tools/Scripts/webkitperl](https://trac.webkit.org/browser/webkit/trunk/Tools/Scripts/webkitperl).
203
Use `Tools/Scripts/test-webkitperl` to run these tests.
204
205
## Performance Testing in WebKit
206
207
The WebKit project has a "no performance regression" policy.
208
We maintain the performance of the following of the benchmarks and are located under [PerformanceTests](https://trac.webkit.org/browser/webkit/trunk/PerformanceTests).
209
If your patch regresses one of these benchmarks even slightly (less than 1%), it will get reverted.
210
211
* **JetStream2** - Measures JavaScript and WASM performance.
212
* **MotionMark** - Measures graphics performance.
213
* **Speedometer 2** - Measures WebKit’s performance for complex web apps.
214
215
The following are benchmarks maintained by Apple's WebKit team but not available to other open source contributors
216
since Apple doesn't have the right to redistribute the content.
217
If your WebKit patch regresses one of these tests, your patch may still get reverted.
218
219
* **RAMification** - Apple's internal JavaScript memory benchmark.
220
* **ScrollPerf** - Apple's internal scrolling performance tests.
221
* **PLT** - Apple's internal page load time tests.
222
* **Membuster / PLUM** - Apple's internal memory tests. Membuster for macOS and PLUM for iOS and iPadOS.
223
224
## Contributing code to WebKit
225
226
WebKit has a rigorous code contribution process and policy in place to maintain the quality of code.
227
228
### Coding style
229
230
Code you write must follow WebKit’s [coding style guideline](https://webkit.org/contributing-code/#code-style-guidelines).
231
You can run `Tools/Scripts/check-webkit-style` to check whether your code follows the coding guidelines or not
232
(it can report false positives or false negatives).
233
If you use `Tools/Scripts/webkit-patch upload` to upload your patch,
234
it automatically runs the style checker against the code you changed so there is no need to run `check-webkit-style` separately.
235
236
Some older parts of the codebase do not follow these guidelines.
237
If you are modifying such code, it is generally best to clean it up to comply with the current guidelines.
238
239
### Convenience Tools
240
241
`Tools/Scripts/webkit-patch` provides a lot of utility functions like applying the latest patch on [bugs.webkit.org](https://bugs.webkit.org/) (`apply-from-bug`)
242
and uploading a patch (`upload --git-commit=<commit hash>`) to a [bugs.webkit.org](https://bugs.webkit.org/) bug.
243
Use `--all-commands` to the list of all commands this tool supports.
244
245
### Licensing
246
247
Much of the code we inherited from [KHTML](https://en.wikipedia.org/wiki/KHTML) is licensed under [LGPL](https://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License).
248
New code contributed to WebKit will use the [two clause BSD license](https://trac.webkit.org/browser/trunk/Source/WebCore/LICENSE-APPLE).
249
When contributing new code, update the copyright date.
250
When moving the existing code, you need to include the original copyright notice for the moved code
251
and you should also not change the license, which may be BSD or LGPL depending on a file, without the permission of the copyright holders.
252
253
### Regression Tests
254
255
Once you have made a code change, you need to run the aforementioned tests (layout tests, API tests, etc...)
256
to make sure your code change doesn’t break existing functionality.
257
These days, uploading a patch on [bugs.webkit.org](https://bugs.webkit.org/) triggers the Early Warning System (a.k.a. EWS)
258
259
For any bug fix or a feature addition, there should be a new test demonstrating the behavior change caused by the code change.
260
If no such test can be written in a reasonable manner (e.g. the fix for a hard-to-reproduce race condition),
261
then the reason writing a tests is impractical should be explained in the accompanying change log.
262
263
Any patch which introduces new test failures or performance regressions may be reverted.
264
It’s in your interest to wait for the Early Warning System to fully build and test your patch on all relevant platforms.
265
266
### ChangeLog Files
267
268
ChangeLogs are simple text files which provide historical documentation for all changes to the WebKit project.
269
All patches require an entry to the ChangeLog.
270
271
The first line contains the date, your full name, and your email address.
272
Use this to write up a brief summary of the changes you’ve made.
273
Don’t worry about the “Reviewed by NOBODY (OOPS!)” line, the person landing your patch will fill this in.
274
There is one ChangeLog per top-level directory.
275
If you changed code and tests you will need to edit at least two ChangeLogs.
276
`Tools/Scripts/prepare-ChangeLog` script will create stub entries for ChangeLog files based on code changes you made in your Git or Subversion checkouts.
277
278
You should edit these stubs to describe your change, including the full URL to the bug (example entry, note that you can use `--bug` flag).
279
(You should set `EMAIL_ADDRESS` and `CHANGE_LOG_NAME` in your environment if you will be running this script frequently.)
280
A typical change log entry before being submitted to [bugs.webkit.org](https://bugs.webkit.org/) looks like this:
281
282
```
283
2012-10-04 Enrica Casucci <e•••••@apple.com>
284
285
Font::glyphDataAndPageForCharacter doesn't account for text orientation when using systemFallback on a cold cache.
287
288
Reviewed by NOBODY (OOPS!).
289
290
The text orientation was considered only when there is a cache hit.
291
This change moves the logic to handle text orientation to a separate
292
inline function that is called also when the glyph is added to the cache.
293
294
Test: fast/text/vertical-rl-rtl-linebreak.html
295
296
* platform/graphics/FontFastPath.cpp:
297
(WebCore::applyTextOrientationForCharacter): Added.
298
(WebCore::Font::glyphDataAndPageForCharacter): Modified to use the new function in
299
both cases of cold and warm cache.
300
```
301
302
303
The “No new tests. (OOPS!)” line appears if `prepare-ChangeLog` did not detect the addition of new tests.
304
If your patch does not require test cases (or test cases are not possible), remove this line and explain why you didn’t write tests.
305
Otherwise all changes require test cases which should be mentioned in the ChangeLog.
306
307
## WebKit’s Build System
308
309
Apple’s macOS, iOS, watchOS, and tvOS ports use Xcode and the rest use [CMake](https://en.wikipedia.org/wiki/CMake) to build WebKit.
310
There is an ongoing effort to make Apple's ports also use CMake.
311
312
In order to reduce the compilation time, which used to take 40+ minutes on the fully loaded 2018 15“ MacBook Pro,
313
we bundle up multiple C++ translation units (.cpp files) and compile them as a single translation unit.
314
We call this mechanism *Unified Sources* or *Unified Builds*.
315
316
Unified sources are generated under `WebKitBuild/X/DerivedSources` where X is the name of build configuration such as `Debug` and `Release-iphonesimulator`.
317
For example, `WebKitBuild/Debug/DerivedSources/WebCore/unified-sources/UnifiedSource116.cpp` may look like this:
318
319
```cpp
320
#include "dom/Document.cpp"
321
#include "dom/DocumentEventQueue.cpp"
322
#include "dom/DocumentFragment.cpp"
323
#include "dom/DocumentMarkerController.cpp"
324
#include "dom/DocumentParser.cpp"
325
#include "dom/DocumentSharedObjectPool.cpp"
326
#include "dom/DocumentStorageAccess.cpp"
327
#include "dom/DocumentType.cpp"
328
```
329
330
### How to add a new .h or .cpp file
331
332
To add a new header file or a translation unit (e.g. `.cpp`, `.m`, or `.mm`),
333
open WebKit.xcworkspace and add respective files in each directory.
334
335
Make sure to uncheck the target membership so that it’s not compiled as a part of the framework in xcodebuild.
336
Instead, add the same file in Sources.txt file that exists in each subdirectory of Source.
337
e.g. [Source/WebCore/Sources.txt](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/Sources.txt) for WebCore.
338
This will ensure the newly added file is compiled as a part of *unified sources*.
339
![Screenshot of adding a file to Xcode](resources/xcode-add-file.png)
340
When a header file in WTF is used in WebCore, or a header file in WebCore is used in WebKit or WebKitLegacy,
341
we need to export the file to those projects.
342
To do that, turn on the target membership in respective framework as set the membership to “Private” as seen below.
343
This will ensure the relevant header file is exported from WTF / WebCore to other downstream projects like WebKitLegacy.
344
![Screenshot of exporting a header file](resources/xcode-export-header.png)
345
346
FIXME: Mention WTF_EXPORT_PRIVATE and WEBCORE_EXPORT.
347
348
FIXME: Add instructions on how to add files to CMake.
349
350
### Build Failures with Unified Sources
351
352
Because of Unified Sources, it’s possible that adding a new file will cause a new build failure on some platform.
353
This happens because if `UnifiedSource1.cpp` contains `a.cpp`, `b.cpp`, `c.cpp`, then `#include` in `a.cpp` could have pulled in some header files that `c.cpp` needed.
354
When you add `b2.cpp`, and `c.cpp` moves to `UnifiedSource2.cpp`, `c.cpp` no longer benefits from `a.cpp` “accidentally” satisfying `c.cpp`’s header dependency.
355
When this happens, you need to add a new `#include` to `c.cpp` as it was supposed to be done in the first place.
356
357
### Conditional Compilation
358
359
Every translation unit in WebKit starts by including “config.h”.
360
This file defines a set of [C++ preprocessor macros](https://en.cppreference.com/w/cpp/preprocessor)
361
used to enable or disable code based on the target operating system, platform, and whether a given feature is enabled or disabled.
362
363
For example, the following `#if` condition says that the code inside of it is only compiled if
365
366
```cpp
367
#if ENABLE(SERVICE_WORKER)
368
...
369
#endif
370
```
371
372
Similarly, the following `#if` condition will enable the in-between code only on macOS:
373
374
```cpp
375
#if PLATFORM(MAC)
376
...
377
#endif
378
```
379
380
For code which should be enabled in iOS, watchOS, tvOS, and Mac Catalyst we use `PLATFORM(IOS_FAMILY)`.
381
For each specific variant of iOS family, we also have `PLATFORM(IOS)`, `PLATFORM(WATCHOS)`, `PLATFORM(APPLETV)`, and `PLATFORM(MACCATALYST)`.
382
383
The following `#if` condition will enable the in-between code only if CoreGraphics is used:
384
385
```cpp
386
#if USE(CG)
387
...
388
#endif
389
```
390
391
Finally, if a certain piece of code should only be enabled in an operating system newer than some version,
392
we use `__IPHONE_OS_VERSION_MIN_REQUIRED` or `__MAC_OS_X_VERSION_MIN_REQUIRED`.
393
For example, the following #if enables the in-between code only on macOS 10.14 (macOS Mojave) or above:
394
395
```cpp
396
#if PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED >= 101400
397
...
398
#endif
399
```
400
401
## WebKit’s Continuous Integration Infrastructure
402
403
WebKit’s CI ([continuous integration](https://en.wikipedia.org/wiki/Continuous_integration)) infrastructure is located at [build.webkit.org](http://build.webkit.org/)).
404
405
[build.webkit.org](http://build.webkit.org/) will build and test commits from WebKit in the chronological order
406
and report test results to [results.webkit.org](http://results.webkit.org/).
407
Due to the chronological ordering, results could be a few hours behind during the work week.
408
409
410
We also have a dashboard to monitor the health of [build.webkit.org](http://build.webkit.org/)
411
at [build.webkit.org/dashboard](https://build.webkit.org/dashboard/).
412
If you observe that some bots are offline, or otherwise not processing your patch,
413
please notify [webkit-dev@webkit.org](mailto:webkit-dev@webkit.org).
414
415
This dashboard isn't great for investigating individual test failures,
416
[results.webkit.org](http://results.webkit.org/) is a better tool for such investigations.
417
It keeps track of individual test status by configuration over time.
418
You can search individual tests by name or look at the historical results of entire test suites.
419
These results will link back to the test runs in buildbot which are associated with a specific failure.
420
See layout tests section for more details on how to use these tools to investigate test failures observed on bots.
421
422
FIXME: Add a section about downloading build products from build.webkit.org.
423
424
# Memory Management in WebKit
425
426
In WebKit, when an object is owned by another object,
427
we typically use [`std::unique_ptr`](https://en.cppreference.com/w/cpp/memory/unique_ptr) to express that ownership.
428
WebKit uses two primary management strategies when objects in other cases:
430
431
## Garbage collection in WebKit
432
433
FIXME: Write this.
434
435
## Reference counting in WebKit
436
437
### Overview
438
439
Most of WebCore objects are not managed by JavaScriptCore’s garbage collector.
440
Instead, we use [reference counting](https://en.wikipedia.org/wiki/Reference_counting).
441
We have two referencing counting pointer types:
444
RefPtr is intended to behave like a C++ pointer whereas Ref is intended to behave like a C++ reference,
445
meaning that the former can be set to `nullptr` but the latter cannot.
446
447
```cpp
448
Ref<A> a1; // This will result in compilation error.
449
RefPtr<A> a2; // This is okay.
450
Ref<A> a3 = A::create(); // This is okay.
451
a3->f(); // Calls f() on an instance of A.
452
A* a4 = a3.ptr();
453
a4 = a2.get();
454
```
455
456
457
Unlike C++‘s[`std::shared_ptr`](https://en.cppreference.com/w/cpp/memory/shared_ptr),
458
the implementation of referencing counting is a part of a managed object.
459
The requirements for an object to be used with `RefPtr` and `Ref` is as follows:
460
461
* It implements `ref()` and `deref()` member functions
462
* Each call to `ref()` and `deref()` will increment and decrement its internal reference counter
463
* The initial call to `ref()` is implicit in `new`,
464
after the object had been allocated and the constructor has been called upon;
465
i.e. meaning that the reference count starts at 1.
466
* When `deref()` is called when its internal reference counter reaches 0, “this” object is destructed and deleted.
467
468
There is a convenience super template class,
470
which implements this behavior for any inherited class T automatically.
471
472
### How to use RefPtr and Ref
473
474
When an object which implements the semantics required by RefPtr and Ref is created via new,
475
we must immediately *adopt* it into `Ref` type using `adoptRef` as follows:
476
477
```cpp
478
class A : public RefCounted<T> {
479
public:
480
int m_foo;
481
482
int f() { return m_foo; }
483
484
static Ref<A> create() { return adoptRef(*new A); }
485
private:
486
A() = default;
487
};
488
```
489
490
This will create an instance of `Ref` without calling `ref()` on the newly created object, avoiding the unnecessary increment from 0 to 1.
491
WebKit’s coding convention is to make the constructor private and add a static `create` function
492
which returns an instance of a ref counted object after adopting it.
493
494
Note that returning RefPtr or Ref is efficient thanks to [copy elision](https://en.cppreference.com/w/cpp/language/copy_elision) in C++11,
495
and the following example does not create a temporary Ref object using copy constructor):
496
497
```cpp
498
Ref<A> a = A::create();
499
```
500
501
When passing the ownership of a ref-counted object to a function,
502
use rvalue reference with `WTFMove` (equivalent to `std::move` with some safety checks),
503
and use a regular reference when there is a guarantee for the caller to keep the object alive as follows:
504
505
```cpp
506
class B {
507
public:
508
void setA(Ref<A>&& a) { m_a = WTFMove(a); }
509
private:
510
Ref<A> m_a;
511
};
512
513
...
514
515
void createA(B& b) {
516
b.setA(A::create());
517
}
518
```
519
520
Note that there is no `WTFMove` on `A::create` due to copy elision.
521
522
### Forwarding ref and deref
523
524
As mentioned above, objects that are managed with `RefPtr` and `Ref` do not necessarily have to inherit from `RefCounted`.
525
One common alternative is to forward `ref` and `deref` calls to another object which has the ownership.
526
For example, in the following example, `Parent` class owns `Child` class.
527
When someone stores `Child` in `Ref` or `RefPtr`, the referencing counting of `Parent` is incremented and decremented on behalf of `Child`.
528
Both `Parent` and `Child` are destructed when the last `Ref` or `RefPtr` to either object goes away.
529
530
```cpp
531
class Parent : RefCounted<Parent> {
532
public:
533
static Ref<Parent> create() { return adoptRef(*new Parent); }
534
535
Child& child() {
536
if (!m_child)
537
m_child = makeUnique<Child>(*this);
538
return m_child
539
}
540
541
private:
542
std::unique_ptr<Child> m_child;
543
};
544
545
class Child {
546
public:
547
ref() { m_parent.ref(); }
548
deref() { m_parent.deref(); }
549
550
private:
551
Child(Parent& parent) : m_parent(parent) { }
552
friend class Parent;
553
554
Parent& m_parent;
555
}
556
```
557
558
### Reference Cycles
559
560
A reference cycle occurs when an object X which holds `Ref` or `RefPtr` to another object Y which in turns owns X by `Ref` or `RefPtr`.
561
For example, the following code causes a trivial memory leak because A holds a `Ref` of B, and B in turn holds `Ref` of the A:
562
563
```cpp
564
class A : RefCounted<A> {
565
public:
566
static Ref<A> create() { return adoptRef(*new A); }
567
B& b() {
568
if (!m_b)
569
m_b = B::create(*this);
570
return m_b.get();
571
}
572
private:
573
Ref<B> m_b;
574
};
575
576
class B : RefCounted<B> {
577
public:
578
static Ref<B> create(A& a) { return adoptRef(*new B(a)); }
579
580
private:
581
B(A& a) : m_a(a) { }
582
Ref<A> m_a;
583
};
584
```
585
586
We need to be particularly careful in WebCore with regards to garbage collected objects
587
because they often keep other ref counted C++ objects alive without having any `Ref` or `RefPtr` in C++ code.
588
It’s almost always incorrect to strongly keep JS value alive in WebCore code because of this.
589
590
### ProtectedThis Pattern
591
592
Because many objects in WebCore are managed by tree data structures,
593
a function that operates on a node of such a tree data structure can end up deleting itself (`this` object).
594
This is highly undesirable as such code often ends up having a use-after-free bug.
595
596
To prevent these kinds of bugs, we often employ a strategy of adding `protectedThis` local variable of `Ref` or `RefPtr` type, and store `this` object as [follows](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/dom/ContainerNode.cpp?rev=251041#L565):
597
598
```cpp
599
ExceptionOr<void> ContainerNode::removeChild(Node& oldChild)
600
{
601
// Check that this node is not "floating".
602
// If it is, it can be deleted as a side effect of sending mutation events.
603
ASSERT(refCount() || parentOrShadowHostNode());
604
605
Ref<ContainerNode> protectedThis(*this);
606
607
// NotFoundError: Raised if oldChild is not a child of this node.
608
if (oldChild.parentNode() != this)
609
return Exception { NotFoundError };
610
611
if (!removeNodeWithScriptAssertion(oldChild, ChildChangeSource::API))
612
return Exception { NotFoundError };
613
614
rebuildSVGExtensionsElementsIfNecessary();
615
dispatchSubtreeModifiedEvent();
616
617
return { };
618
}
619
```
620
621
In this code, the act of removing `oldChild` can execute arbitrary JavaScript and delete `this` object.
622
As a result, `rebuildSVGExtensionsElementsIfNecessary` or `dispatchSubtreeModifiedEvent` might be called
623
after `this` object had already been free’ed if we didn’t have `protectedThis`,
624
which guarantees that this object’s reference count is at least 1
625
(because [Ref’s constructor](https://trac.webkit.org/browser/webkit/trunk/Source/WTF/wtf/Ref.h?rev=250005#L63) increments the reference count by 1).
626
627
This pattern can be used for other objects that need to be *protected* from destruction inside a code block.
629
`childToRemove` was passed in using C++ reference.
630
Because this function is going to remove this child node from `this` container node,
631
it can get destructed while the function is still running.
632
To prevent from having any chance of use-after-free bugs,
633
this function stores it in Ref (`protectedChildToRemove`) which guarantees the object to be alive until the function returns control back to the caller:
634
635
```cpp
636
ALWAYS_INLINE bool ContainerNode::removeNodeWithScriptAssertion(Node& childToRemove, ChildChangeSource source)
637
{
638
Ref<Node> protectedChildToRemove(childToRemove);
639
ASSERT_WITH_SECURITY_IMPLICATION(childToRemove.parentNode() == this);
640
{
641
ScriptDisallowedScope::InMainThread scriptDisallowedScope;
642
ChildListMutationScope(*this).willRemoveChild(childToRemove);
643
}
644
..
645
```
646
647
Also see [Darin’s RefPtr Basics](https://webkit.org/blog/5381/refptr-basics/) for further reading.
648
649
## Weak Pointers in WebKit
650
651
In some cases, it’s desirable to express a relationship between two objects without necessarily tying their lifetime.
652
In those cases, `WeakPtr` is useful. Like [std::weak_ptr](https://en.cppreference.com/w/cpp/memory/weak_ptr),
653
this class creates a non-owning reference to an object. There is a lot of legacy code which uses a raw pointer for this purpose,
654
but there is an ongoing effort to always use WeakPtr instead so do that in new code you’re writing.
655
656
To create a `WeakPtr` to an object, we need to make its class inherit from `CanMakeWeakPtr` as follows:
657
658
```cpp
659
class A : CanMakeWeakPtr<A> { }
660
661
...
662
663
function foo(A& a) {
664
WeakPtr<A> weakA = makeWeakPtr(a);
665
}
666
```
667
668
Dereferencing a `WeakPtr` will return `nullptr` when the referenced object is deleted.
669
Because creating a `WeakPtr` allocates an extra `WeakPtrImpl` object,
670
you’re still responsible to dispose of `WeakPtr` at appropriate time.
671
672
### WeakHashSet
673
674
While ordinary `HashSet` does not support having `WeakPtr` as its elements,
675
there is a specialized `WeakHashSet` class, which supports referencing a set of elements weakly.
676
Because `WeakHashSet` does not get notified when the referenced object is deleted,
677
the users / owners of `WeakHashSet` are still responsible for deleting the relevant entries from the set.
678
Otherwise, WeakHashSet will hold onto `WeakPtrImpl` until `computeSize` is called or rehashing happens.
679
680
# Understanding Document Object Model
681
682
## Introduction
683
685
(often abbreviated as DOM) is the tree data structured resulted from parsing HTML.
686
It consists of one or more instances of subclasses of [Node](https://developer.mozilla.org/en-US/docs/Web/API/Node)
687
and represents the document tree structure. Parsing a simple HTML like this:
688
689
```cpp
690
<!DOCTYPE html>
691
<html>
692
<body>hi</body>
693
</html>
694
```
695
696
Will generate the following six distinct DOM nodes:
697
703
* [Text](https://developer.mozilla.org/en-US/docs/Web/API/Text) with the value of “hi”
704
705
Note that HTMLHeadElement (i.e. `<head>`) is created implicitly by WebKit
706
per the way [HTML parser](https://html.spec.whatwg.org/multipage/parsing.html#parsing) is specified.
707
708
Broadly speaking, DOM node divides into the following categories:
709
712
714
as the name suggests a single HTML, SVG, MathML, or other XML document,
716
It is the very first node in any document that gets created and the very last node to be destroyed.
717
718
Note that a single web [page](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/page/Page.h) may consist of multiple documents
723
Because JavaScript can [open a new window](https://developer.mozilla.org/en-US/docs/Web/API/Window/open)
724
under user gestures and have [access back to its opener](https://developer.mozilla.org/en-US/docs/Web/API/Window/opener),
725
multiple web pages across multiple tabs might be able to communicate with one another via JavaScript API
727
728
## JavaScript Wrappers
729
730
Each DOM node’s behavior is implemented as a C++ class in WebCore.
731
JavaScript API is primarily implemented using [Web IDL](https://heycam.github.io/webidl/),
732
an [interface description language](https://en.wikipedia.org/wiki/Interface_description_language),
733
from which various [JS DOM binding code](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/bindings)
735
for example, under `WebKitBuild/Debug/DerivedSources/WebCore/` for debug builds.
736
For example, C++ implementation of [Node](https://developer.mozilla.org/en-US/docs/Web/API/Node)
738
and its JavaScript interface is implemented by JSNode class,
739
most of which is auto-generated but has some custom bindings code in
741
Similarly, C++ implementation of [Range interface](https://developer.mozilla.org/en-US/docs/Web/API/Range)
743
whilst its JavaScript API is implemented by the auto-generated JSRange class.
744
We call instances of the latter JS* classes *JS wrappers*.
745
746
These JS wrappers exist in what we call a [`DOMWrapperWorld`](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/bindings/js/DOMWrapperWorld.h).
747
Each `DOMWrapperWorld` has its own JS wrapper for each C++ object.
748
As a result, a single C++ object may have multiple JS wrappers in distinct `DOMWrapperWorld`s.
749
The most important `DOMWrapperWorld` is the main `DOMWrapperWorld` which runs the scripts of web pages WebKit loaded
750
while other `DOMWrapperWorld`s are typically used to run code for browser extensions and other code injected by applications that embed WebKit.
751
![Diagram of JS wrappers](resources/js-wrapper.png)
752
JSX.h provides `toJS` functions which creates a JS wrapper for X
753
in a given [global object](https://developer.mozilla.org/en-US/docs/Glossary/Global_object)’s `DOMWrapperWorld`,
754
and toWrapped function which returns the underlying C++ object.
755
For example, `toJS` function for [Node](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/dom/Node.h)
756
is defined in [Source/WebCore/bindings/js/JSNodeCustom.h](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/bindings/js/JSNodeCustom.h).
757
758
When there is already a JS wrapper object for a given C++ object,
759
`toJS` function will find the appropriate JS wrapper in
761
of the given `DOMWrapperWorld`.
762
Because a hash map lookup is expensive, some WebCore objects will inherit from
764
which has an inline pointer to the JS wrapper for the main world if one was already created.
765
766
## JS Wrapper Lifecycle Management
767
768
As a general rule, a JS wrapper object keeps its underlying C++ object alive by means of reference counting
770
from which all JS wrappers in WebCore inherits.
771
However, **C++ objects do not keep their corresponding JS wrapper in each world alive** by the virtue of them staying alive
772
as such a circular dependency will result in a memory leak.
773
774
There are two primary mechanisms to keep JS wrappers alive in WebCore:
775
776
* **Visit Children** - When JavaScriptCore’s garbage collection visits some JS wrapper during
778
visit another JS wrapper or JS object that needs to be kept alive.
779
* **Reachable from Opaque Roots** - Tell JavaScriptCore’s garbage collection that a JS wrapper is reachable
780
from an opaque root which was added to the set of opaque roots during marking phase.
781
782
FIXME: Explain how to add new IDL files and where derived sources are generated.
783
784
### Visit Children
785
786
FIXME: Explain how visit children works.
787
788
### Opaque Roots
789
790
FIXME: Explain how opaque roots work.
791
792
## Inserting or Removing DOM Nodes
793
794
FIXME: Talk about how a node insertion or removal works.
795
796
# Understanding Style and Render Tree
797
798
FIXME: Describe rendering/layers/compositing
799
800
# Security Model of Web
801
803
804
FIXME: Write this section.
805
806
# WebKit2: WebKit’s Multi-Process Architecture
807
808
## Overview
809
810
In order to safeguard the rest of the system and allow the application to remain responsive
811
even if the user had loaded web page that infinite loops or otherwise hangs,
812
the modern incarnation of WebKit uses multi-process architecture.
813
Web pages are loaded in its own *WebContent* process.
814
Multiple WebContent processes can share a browsing session, which lives in a shared network process.
815
In addition to handling all network accesses,
816
this process is also responsible for managing the disk cache and Web APIs that allow websites
817
to store structured data such as [Web Storage API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API)
819
![Diagram of WebKit2's multi-process architecture](webkit2-process-architecture.png)
820
Because a WebContent process can Just-in-Time compile arbitrary JavaScript code loaded from the internet,
821
meaning that it can write to memory that gets executed, this process is tightly sandboxed.
822
It does not have access to any file system unless the user grants an access,
823
and it does not have direct access to the underlying operating system’s [clipboard](https://en.wikipedia.org/wiki/Clipboard_(computing)),
824
microphone, or video camera even though there are Web APIs that grant access to those features.
825
Instead, UI process brokers such requests.
826
827
FIXME: How is IPC setup
828
829
FIXME: How to add / modify an IPC message
830
831
# Layout Tests: Tests of the Web for the Web
832
833
Layout tests are WebKit tests written using Web technology such as HTML, CSS, and JavaScript,
834
and it’s the primary mechanism by which much of WebCore is tested.
835
Relevant layout test should be ran while you’re making code changes to WebCore and before uploading a patch to [bugs.webkit.org](https://bugs.webkit.org/).
836
While [bugs.webkit.org](https://bugs.webkit.org/)’s Early Warning System will build and run tests on a set of configurations,
837
individual patch authors are ultimately responsible for any test failures that their patches cause.
838
839
## Test Files and Expected Files
840
841
### Directory Structure
842
843
[LayoutTests](https://trac.webkit.org/browser/webkit/trunk/LayoutTests) directory is organized by the category of tests.
844
For example, [LayoutTests/accessibility](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/accessibility) contains accessibility related tests,
845
and [LayoutTests/fast/dom/HTMLAnchorElement](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/fast/dom/HTMLAnchorElement) contains
846
tests for [the HTML anchor element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a).
847
848
Any file that ends in `.html`, `.htm`, `.shtml`, `.xhtml`, `.mht`, `.xht`, `.xml`, `.svg`, or `.php` is considered as a test
849
unless it’s preceded with `-ref`, `-notref`, `-expected`, or `-expected-mismatch` (these are used for ref tests; explained later).
850
It’s accompanied by another file of the same name except it ends in `-expected.txt` or `-expected.png`.
851
These are called *expected results* and constitutes the baseline output of a given test.
852
When layout tests are ran, the test runner generates an output in the form of a plain text file and/or an PNG image,
853
and it is compared against these expected results.
854
855
In the case expected results may differ from one platform to another,
856
the expected results for each test is stored in [LayoutTests/platform](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/platform).
857
The expected result of a given test exists in the corresponding directory in
858
each subdirectory of [LayoutTests/platform](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/platform).
859
For example, the expected result of [LayoutTests/svg/W3C-SVG-1.1/animate-elem-46-t.svg](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/svg/W3C-SVG-1.1/animate-elem-46-t.svg)
860
for macOS Mojave is located at [LayoutTests/platform/mac-mojave/svg/W3C-SVG-1.1/animate-elem-46-t-expected.txt](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/platform/mac-mojave/svg/W3C-SVG-1.1/animate-elem-46-t-expected.txt).
861
862
These platform directories have a fallback order.
863
For example, running tests for WebKit2 on macOS Catalina will use the following fallback path from the most specific to most generic:
864
865
* platform/mac-catalina-wk2 - Results for WebKit2 on macOS Catalina.
866
* platform/mac-catalina - Results for WebKit2 and WebKitLegacy on macOS Catalina.
867
* platform/mac-wk2 - Results for WebKit2 on all macOS.
868
* platform/mac - Results for all macOS.
869
* platform/wk2 - Results for WebKit2 on every operating system.
870
* generic - Next to the test file.
871
872
### Imported Tests
873
874
Tests under [LayoutTests/imported](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/imported) are imported from other repositories.
875
**They should not be modified by WebKit patches** unless the change is made in respective repositories first.
876
877
Most notable is [Web Platform Tests](https://web-platform-tests.org/),
878
which are imported under [LayoutTests/imported/w3c/web-platform-tests](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/imported/w3c/web-platform-tests).
879
These are cross browser vendor tests developed by W3C. Mozilla, Google, and Apple all contribute many tests to this shared test repository.
880
881
### HTTP Tests
882
883
FIXME: Explain how to start and open tests that require HTTP server.
884
885
## Test Expectations
886
887
FIXME: Explain how test expectations work.
888
889
## Running Layout Tests
890
891
To run layout tests, use `Tools/Scripts/run-webkit-tests`.
892
It optionally takes file paths to a test file or directory and options on how to run a test.
893
For example, in order to just run `LayoutTests/fast/dom/Element/element-traversal.html`, do:
894
895
```sh
896
Tools/Scripts/run-webkit-tests fast/dom/Element/element-traversal.html
897
```
898
899
Because there are 50,000+ tests in WebKit,
900
you typically want to run a subset of tests that are relevant to your code change
901
(e.g. `LayoutTests/storage/indexeddb/` if you’re working on IndexedDB) while developing the code change,
902
and run all layout tests at the end on your local machine or rely on the Early Warning System on [bugs.webkit.org](https://bugs.webkit.org/) for more thorough testing.
903
904
Specify `--debug` or `--release` to use either release or debug build.
905
To run tests using iOS simulator, you can specify either `--ios-simulator`, `--iphone-simulator`,
906
or `--ipad-simulator` based on whichever simulator is desired.
907
908
By default, `run-webkit-tests` will run all the tests you specified once in the lexicological order of test paths
909
relative to `LayoutTests` directory and retry any tests that have failed.
910
If you know the test is going to fail and don’t want retries, specify `--no-retry-failures`.
911
912
Because there are so many tests, `run-webkit-tests` will runs tests in different directories in parallel
913
(i.e. all tests in a single directory is ran sequentially one after another).
914
You can control the number of parallel test runners using `--child-processes` option.
915
916
`run-webkit-tests` has many options.
917
Use `--help` to enumerate all the supported options.
918
919
### Repeating Layout Tests
920
921
When you’re investigating flaky tests or crashes, it might be desirable to adjust this.
922
`--iterations X` option will specify the number of times the list of tests are ran.
923
For example, if we are running tests A, B, C and `--iterations 3` is specified,
924
`run-webkit-tests` will run: A, B, C, A, B, C, A, B, C.
925
Similarly, `--repeat-each` option will specify the number of times each test is repeated before moving onto next test.
926
For example, if we’re running tests A, B, C, and `--repeat-each 3` is specified, `run-webkit-tests` will run: A, A, A, B, B, B, C, C, C.
927
`--exit-after-n-failures` option will specify the total number of test failures before `run-webkit-tests` will stop.
928
In particular, `--exit-after-n-failures=1` is useful when investigating a flaky failure
929
so that `run-webkit-tests` will stop when the failure actually happens for the first time.
930
931
### Test Results
932
933
Whenever tests do fail, run-webkit-tests will store results in `WebKitBuild/Debug/layout-test-results`
934
mirroring the same directory structure as `LayoutTests`.
935
For example, the actual output produced for `LayoutTests/editing/inserting/typing-001.html`,
936
if failed, will appear in `WebKitBuild/Debug/layout-test-results/editing/inserting/typing-001-actual.txt`.
937
run-webkit-tests also generates a web page with the summary of results in
938
`WebKitBuild/Debug/layout-test-results/results.html` and automatically tries to open it in Safari using the local build of WebKit.
939
940
> If Safari fails to launch, specify `--no-show-results` and open results.html file manually.
941
942
### Updating Expected Results
943
944
If you’ve updated a test content or test’s output changes with your code change (e.g. more test case passes),
945
then you may have to update `-expected.txt` file accompanying the test.
946
To do that, first run the test once to make sure the diff and new output makes sense in results.html,
947
and run the test again with `--reset-results`.
948
This will update the matching `-expected.txt` file.
949
950
You may need to manually copy the new result to other -expected.txt files that exist under `LayoutTests` for other platforms and configurations.
951
Find other `-expected.txt` files when you’re doing this.
952
953
When a new test is added, `run-webkit-tests` will automatically generate new `-expected.txt` file for your test.
954
You can disable this feature by specifying `--no-new-test-results` e.g. when the test is still under development.
955
956
## Different Styles of Layout Tests
957
958
There are multiple styles of layout tests in WebKit.
959
960
### **Render tree dumps**
961
962
This is the oldest style of layout tests, and the default mode of layout tests.
963
It’s a text serialization of WebKit’s render tree and its output looks like
965
966
```
967
layer at (0,0) size 800x600
968
RenderView at (0,0) size 800x600
969
layer at (0,0) size 800x600
970
RenderBlock {HTML} at (0,0) size 800x600
971
RenderBody {BODY} at (8,8) size 784x584
972
RenderInline {A} at (0,0) size 238x18 [color=#0000EE]
973
RenderInline {B} at (0,0) size 238x18
974
RenderText {#text} at (0,0) size 238x18
975
text run at (0,0) width 238: "the second copy should not be bold"
976
RenderText {#text} at (237,0) size 5x18
977
text run at (237,0) width 5: " "
978
RenderText {#text} at (241,0) size 227x18
979
text run at (241,0) width 227: "the second copy should not be bold"
980
```
981
982
This style of layout tests is discouraged today because its outputs are highly dependent on each platform,
983
and end up requiring a specific expected result in each platform.
984
But they’re still useful when testing new rendering and layout feature or bugs thereof.
985
986
These tests also have accompanying `-expected.png` files but `run-webkit-tests` doesn't check the PNG output against the expected result by default.
987
To do this check, pass `--pixel`.
988
Unfortunately, many *pixel tests* will fail because we have not been updating the expected PNG results a good chunk of the last decade.
989
However, these pixel results might be useful when diagnosing a new test failure.
990
For this reason, `run-webkit-tests` will automatically generate PNG results when retrying the test,
991
effectively enabling `--pixel` option for retries.
992
993
### dumpAsText test
994
995
These are tests that uses the plain text serialization of the test page as the output (as if the entire page’s content is copied as plain text).
996
All these tests call `testRunner.dumpAsText` to trigger this behavior.
997
The output typically contains a log of text or other informative output scripts in the page produced.
998
For example, [LayoutTests/fast/dom/anchor-toString.html](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/fast/dom/anchor-toString.html) is written as follows:
999
1000
```html
1001
<a href="http://localhost/sometestfile.html" id="anchor">
1002
A link!
1003
</a>
1004
<br>
1005
<br>
1006
<script>
1007
{
1008
if (window.testRunner)
1009
testRunner.dumpAsText();
1010
1011
var anchor = document.getElementById("anchor");
1012
document.write("Writing just the anchor object - " + anchor);
1013
1014
var anchorString = String(anchor);
1015
document.write("<br><br>Writing the result of the String(anchor) - " + anchorString);
1016
1017
var anchorToString = anchor.toString();
1018
document.write("<br><br>Writing the result of the anchor's toString() method - " + anchorToString);
1019
}
1020
</script>
1021
```
1022
1024
1025
```
1026
A link!
1027
1028
Writing just the anchor object - http://localhost/sometestfile.html
1029
1030
Writing the result of the String(anchor) - http://localhost/sometestfile.html
1031
1032
Writing the result of the anchor's toString() method - http://localhost/sometestfile.html
1033
```
1034
1035
### js-test.js and js-test-pre.js tests
1036
1037
These are variants of dumpAsText test which uses WebKit’s assertion library:
1040
It consists of shouldX function calls which takes two JavaScript code snippet which are then executed and outputs of which are compared.
1041
js-test.js is simply a new variant of js-test-pre.js that doesn’t require
1042
the inclusion of [LayoutTests/resources/js-test-post.js](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/resources/js-test-post.js) at the end.
1043
**Use js-test.js in new tests**, not js-test-pre.js.
1044
1045
For example, [LayoutTests/fast/dom/Comment/remove.html](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/fast/dom/Comment/remove.html)
1047
on [Comment node](https://developer.mozilla.org/en-US/docs/Web/API/Comment) is written as:
1048
1049
```html
1050
<!DOCTYPE html>
1051
<script src="../../../resources/js-test-pre.js"></script>
1052
<div id="test"></div>
1053
<script>
1054
1055
description('This tests the DOM 4 remove method on a Comment.');
1056
1057
var testDiv = document.getElementById('test');
1058
var comment = document.createComment('Comment');
1059
testDiv.appendChild(comment);
1060
shouldBe('testDiv.childNodes.length', '1');
1061
comment.remove();
1062
shouldBe('testDiv.childNodes.length', '0');
1063
comment.remove();
1064
shouldBe('testDiv.childNodes.length', '0');
1065
1066
</script>
1067
<script src="../../../resources/js-test-post.js"></script>
1068
```
1069
1071
1072
```
1073
This tests the DOM 4 remove method on a Comment.
1074
1075
On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
1076
1077
1078
PASS testDiv.childNodes.length is 1
1079
PASS testDiv.childNodes.length is 0
1080
PASS testDiv.childNodes.length is 0
1081
PASS successfullyParsed is true
1082
1083
TEST COMPLETE
1084
```
1085
1086
`description` function specifies the description of this test, and subsequent shouldBe calls takes two strings,
1087
both of which are evaluated as JavaScript and then compared.
1088
1089
Some old js-test-pre.js tests may put its test code in a separate JS file but we don’t do that anymore to keep all the test code in one place.
1090
1092
Here are some examples:
1093
1094
* `debug(msg)` - Inserts a debug / log string in the output.
1095
* `evalAndLog(code)` - Similar to `debug()` but evaluates code as JavaScript.
1096
* `shouldNotBe(a, b)` - Generates `PASS` if the results of evaluating `a` and `b` differ.
1097
* `shouldBeTrue(code)` - Shorthand for `shouldBe(code, 'true')`.
1098
* `shouldBeFalse(code)` - Shorthand for `shouldBe(code, 'false')`.
1099
* `shouldBeNaN(code)` - Shorthand for `shouldBe(code, 'NaN')`.
1100
* `shouldBeNull(code)` - Shorthand for `shouldBe(code, 'null')`.
1101
* `shouldBeZero(code)` - Shorthand for `shouldBe(code, '0')`.
1102
* `shouldBeEqualToString(code, string)` - Similar to `shouldBe` but the second argument is not evaluated as string.
1103
* `finishJSTest()` - When js-test.js style test needs to do some async work, define the global variable named jsTestIsAsync and set it to true. When the test is done, call this function to notify the test runner (don’t call `testRunner.notifyDone` mentioned later directly). See [an example](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/fast/dom/iframe-innerWidth.html).
1104
1105
**It’s important to note that these shouldX functions only add output strings that say PASS or FAIL. If the expected result also contains the same FAIL strings, then run-webkit-tests will consider the whole test file to have passed.**
1106
1107
Another way to think about this is that `-expected.txt` files are baseline outputs, and baseline outputs can contain known failures.
1108
1109
There is a helper script to create a template for a new js-test.js test. The following will create new test named `new-test.html` in [LayoutTests/fast/dom](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/fast/dom):
1110
1111
```sh
1112
Tools/Scripts/make-new-script-test fast/dom/new-test.html
1113
```
1114
1115
### dump-as-markup.js Tests
1116
1117
A dump-as-markup.js test is yet another variant of dumpAsText test,
1118
which uses [LayoutTests/resources/dump-as-markup.js](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/resources/dump-as-markup.js).
1119
This style of test is used when it’s desirable to compare the state of the DOM tree before and after some operations.
1120
For example, many tests under [LayoutTests/editing](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/editing)
1121
use this style of testing to test complex DOM mutation operations such as pasting HTML from the users’ clipboard.
1122
dump-as-markup.js adds `Markup` on the global object and exposes a few helper functions.
1123
Like js-test.js tests, a test description can be specified via `Markup.description`.
1124
The test then involves `Markup.dump(node, description)` to serialize the state of DOM tree as plain text
1125
where `element` is either a DOM [node](https://developer.mozilla.org/en-US/docs/Web/API/Node)
1126
under which the state should be serialized or its [id](https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/id).
1127
1128
For example, [LayoutTests/editing/inserting/insert-list-in-table-cell-01.html](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/editing/inserting/insert-list-in-table-cell-01.html) is written as follows:
1129
1130
```html
1131
<!DOCTYPE html>
1132
<div id="container" contenteditable="true"><table border="1"><tr><td id="element">fsdf</td><td>fsdf</td></tr><tr><td>gghfg</td><td>fsfg</td></tr></table></div>
1133
<script src="../editing.js"></script>
1134
<script src="../../resources/dump-as-markup.js"></script>
1135
<script>
1136
Markup.description('Insert list items in a single table cell:');
1137
1138
var e = document.getElementById("element");
1139
setSelectionCommand(e, 0, e, 1);
1140
Markup.dump('container', 'Before');
1141
1142
document.execCommand("insertOrderedList");
1143
Markup.dump('container', 'After');
1144
</script>
1145
```
1146
1148
1149
```
1150
Insert list items in a single table cell:
1151
1152
Before:
1153
| <table>
1154
| border="1"
1155
| <tbody>
1156
| <tr>
1157
| <td>
1158
| id="element"
1159
| "<#selection-anchor>fsdf<#selection-focus>"
1160
| <td>
1161
| "fsdf"
1162
| <tr>
1163
| <td>
1164
| "gghfg"
1165
| <td>
1166
| "fsfg"
1167
1168
After:
1169
| <table>
1170
| border="1"
1171
| <tbody>
1172
| <tr>
1173
| <td>
1174
| id="element"
1175
| <ol>
1176
| <li>
1177
| "<#selection-anchor>fsdf<#selection-focus>"
1178
| <br>
1179
| <td>
1180
| "fsdf"
1181
| <tr>
1182
| <td>
1183
| "gghfg"
1184
| <td>
1185
| "fsfg"
1186
```
1187
1188
### testharness.js Tests
1189
1190
This is yet another variant of dumpAsText test which uses the test harness of [Web Platform Tests](https://web-platform-tests.org/),
1191
which is [W3C](https://www.w3.org/)’s official tests for the Web.
1192
There is an [extensive documentation](https://web-platform-tests.org/writing-tests/testharness-api.html) on how this harness works.
1193
1194
> As mentioned above, do not modify tests in [LayoutTests/imported/w3c/web-platform-tests](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/imported/w3c/web-platform-tests)
1195
unless the same test changes are made in Web Platform Tests’ primary repository.
1196
1197
### Reference Tests
1198
1199
Reference tests are special in that they don’t have accompanying `-expected.txt` files.
1200
Instead, they have a matching or mismatching expected result file.
1201
Both the test file and the accompanying matching or mismatching expected result generate PNG outputs.
1202
The test passes if the PNG outputs of the test and the matching expected result are the same; the test fails otherwise.
1203
For a test with a mismatching expected result, the test passes if the PNG outputs of the test and the mismatching expected result are not the same, and fails if they are the same.
1204
1205
A matching expected result or a mismatching expected result can be specified in a few ways:
1206
1207
* The file with the same name as the test name except it ends with `-expected.*` or `-ref.*` is a matching expected result for the test.
1208
* The file with the same name as the test name except it ends with `-expected-mismatch.*` or `-notref.*` is a matching expected result for the test.
1209
* The file specified by a HTML link element in the test file with `match` relation: `<link rel=match href=X>` where X is the relative file path is a matching expected result.
1210
* The file specified by a HTML link element in the test file with `mismatch` relation: `<link rel=mismatch href=X>` where X is the relative file path is a mismatching expected result.
1211
1212
For example, [LayoutTests/imported/w3c/web-platform-tests/2dcontext/line-styles/lineto_a.html](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/imported/w3c/web-platform-tests/2dcontext/line-styles/lineto_a.html) specifies [lineto_ref.html](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/imported/w3c/web-platform-tests/2dcontext/line-styles/lineto_ref.html) in the same directory as the matching expected result as follows:
1213
1214
```html
1215
<!DOCTYPE html>
1216
<meta charset=utf-8>
1217
<link rel=match href=lineto_ref.html>
1218
<style>
1219
html, body {
1220
margin: 0;
1221
padding: 0;
1222
}
1223
</style>
1224
<canvas id="c" width="150" height="150" >
1225
Your browser does not support the HTML5 canvas tag.</canvas>
1226
1227
<script>
1228
var c = document.getElementById("c");
1229
var ctx = c.getContext("2d");
1230
1231
ctx.beginPath();
1232
ctx.moveTo(20, 20);
1233
ctx.lineTo(20, 130);
1234
ctx.lineTo(130, 130);
1235
ctx.lineTo(130, 20);
1236
ctx.closePath();
1237
1238
ctx.fillStyle = '#90EE90';
1239
ctx.fill();
1240
</script>
1241
```
1242
1243
## Test Runners
1244
1245
Most layout tests are designed to be runnable inside a browser but run-webkit-tests uses a special program to run them.
1246
Our continuous integration system as well as the Early Warning System uses run-webkit-tests to run layout tests.
1247
In WebKit2, this is appropriately named [WebKitTestRunner](https://trac.webkit.org/browser/webkit/trunk/Tools/WebKitTestRunner).
1248
In WebKit1 or WebKitLegacy, it’s [DumpRenderTree](https://trac.webkit.org/browser/webkit/trunk/Tools/DumpRenderTree),
1249
which is named after the very first type of layout tests, which generated the text representation of the render tree.
1250
1251
### Extra Interfaces Available in Test Runners
1252
1253
Both WebKitTestRunner and DumpRenderTree expose a few extra interfaces to JavaScript on `window` (i.e. global object) in order to emulate user inputs,
1254
enable or disable a feature, or to improve the reliability of testing.
1255
1257
- `GCController.collect()` triggers a synchronous full garbage collection.
1258
This function is useful for testing crashes or erroneous premature collection of JS wrappers and leaks.
1260
- TestRunner interface exposes many methods to control the behaviors of WebKitTestRunner and DumpRenderTree.
1261
Some the most commonly used methods are as follows:
1262
* `waitUntilDone()` / `notifyDone()` - These functions are useful when writing tests that involve asynchronous tasks
1263
which may require the test to continue running beyond when it finished loading.
1264
`testRunner.waitUntilDone()` makes WebKitTestRunner and DumpRenderTree not end the test when a layout test has finished loading.
1265
The test continues until `testRunner.notifyDone()` is called.
1266
* `dumpAsText(boolean dumpPixels)` - Makes WebKitTestRunner and DumpRenderTree output the plain text of the loaded page instead of the state of the render tree.
1267
* `overridePreference(DOMString preference, DOMString value)` - Overrides WebKit’s preferences.
1268
For WebKit2, these preferences are defined in [Source/WebKit/Shared/WebPreferences.yaml](https://trac.webkit.org/browser/webkit/trunk/Source/WebKit/Shared/WebPreferences.yaml).
1269
For WebKitLegacy, these are defined in [Source/WebKitLegacy/mac/WebView/WebPreferences.h](https://trac.webkit.org/browser/webkit/trunk/Source/WebKitLegacy/mac/WebView/WebPreferences.h) for macOS
1270
and [Source/WebKitLegacy/win/WebPreferences.h](https://trac.webkit.org/browser/webkit/trunk/Source/WebKitLegacy/win/WebPreferences.h) for Windows.
1272
- Exposes methods to emulate mouse, keyboard, and touch actions.
1273
**Use [ui-helpers.js](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/resources/ui-helper.js) script** instead of directly calling methods on this function.
1274
This will ensure the test will be most compatible with all the test configurations we have.
1276
- Exposes methods to emulate user inputs like eventSender mostly on iOS WebKit2.
1277
**Use [ui-helpers.js](https://trac.webkit.org/browser/webkit/trunk/LayoutTests/resources/ui-helper.js) script** instead of directly calling methods on this function.
1278
This will ensure the test will be most compatible with all the test configurations we have.
1280
- Exposes methods to test [input methods](https://en.wikipedia.org/wiki/Input_method).
1281
1282
Additionally, [WebCore/testing](https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/testing) exposes a few testing hooks to test its internals:
1283
1285
- Exposes various hooks into WebCore that shouldn’t be part of WebKit or WebKitLegacy API.
1287
- Exposes various WebCore settings and let tests override them.
1288
Note that WebKit layer code depends on [Source/WebKit/Shared/WebPreferences.yaml](https://trac.webkit.org/browser/webkit/trunk/Source/WebKit/Shared/WebPreferences.yaml),
1289
and will not respect this override.
1290
Because of this, it’s preferable to override the equivalent preference via `testRunner.overridePreference`
1291
unless you know for sure WebKit or WebKitLegacy layer of code isn’t affected by the setting you’re overriding.
1292
1293
### Enabling or Disabling a Feature in Test Runners
1294
1295
FIXME: Mention test-runner-options
1296
1297
## Test Harness Scripts
1298
1299
FIXME: Write about dump-as-markup.js, and ui-helper.js
1300
1301
## Investigating Test Failures Observed on Bots
1302
1303
There are multiple tools to investigate test failures happening on our continuous integration system
1304
([build.webkit.org](http://build.webkit.org/)).
1305
The most notable is flakiness dashboard:
1306
[results.webkit.org](https://results.webkit.org/)
1307
1308
FIXME: Write how to investigate a test failure.
1309
1310
## Debugging Layout Tests in Xcode
1311
1312
The easiest way to debug a layout test is with WebKitTestRunner or DumpRenderTree.
1313
In Product > Scheme, select “All Source”.
1314
1315
In Product > Scheme > Edit Scheme, open “Run” tab.
1316
Pick WebKitTestRunner or DumpRenderTree, whichever is desired in “Executable”.
1317
1318
![Screenshot of specifying DumpRenderTree as the target of "Run" scheme](resources/xcode-scheme-dumprendertree.png)
1319
Go to Arguments and specify the path to the layout tests being debugged relative to where the build directory is located.
1320
e.g. `../../LayoutTests/fast/dom/Element/element-traversal.html` if `WebKitBuild/Debug` is the build directory.
1321
![Screenshot of Xcode specifying a layout test in an argument to DumpRenderTree](resources/xcode-scheme-layout-test.png)
1322
You may want to specify OS_ACTIVITY_MODE environmental variable to “disable”
1323
in order to suppress all the system logging that happens during the debugging session.
1324
1325
You may also want to specify `--no-timeout` option to prevent WebKitTestRunner or DumpRenderTree
1326
to stop the test after 30 seconds if you’re stepping through code.
1327
1328
Once this is done, you can run WebKitTestRunner or DumpRenderTree by going to Product > Perform Action > Run without Building.
1329
1330
Clicking on “Run” button may be significantly slower due to Xcode re-building every project and framework each time.
1331
You can disable this behavior by going to “Build” tab and unchecking boxes for all the frameworks involved for “Run”:
1332
![Screenshot of Xcode unchecking build options for all but DumpRenderTree for "Run" scheme](resources/xcode-build-settings-for-run.png)
1333
1334
### Attaching to WebContent Process
1335
1336
You may find Xcode fails to attach to WebContent or Networking process in the case of WebKitTestRunner.
1337
In those cases, attach a breakpoint in UIProcess code
1338
such as [`TestController::runTest` in WebKitTestRunner right before `TestInvocation::invoke` is called](https://trac.webkit.org/browser/webkit/trunk/Tools/WebKitTestRunner/TestController.cpp?rev=252228#L1701).
1339
Once breakpoint is hit in the UIProcess, attach to `WebContent.Development` or `Networking.Development` process manually in Xcode via Debug > Attach to Process.
1340
1341
# Dive into API tests
1342
1343
FIXME: Talk about how to debug API tests.
1344
1345
# Logging in WebKit
1346
1347
FIXME: Write this.
1348