https://github.com/mozilla/gecko-dev
Tip revision: 00a764a20bbcd6c3d64c538175f2e407fb81d835 authored by ffxbld on 04 May 2015, 19:30:19 UTC
Added FIREFOX_31_7_0esr_RELEASE FIREFOX_31_7_0esr_BUILD1 tag(s) for changeset b25163dd3c90. DONTBUILD CLOSED TREE a=release
Added FIREFOX_31_7_0esr_RELEASE FIREFOX_31_7_0esr_BUILD1 tag(s) for changeset b25163dd3c90. DONTBUILD CLOSED TREE a=release
Tip revision: 00a764a
jsgc.cpp
/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 4 -*-
* vim: set ts=8 sts=4 et sw=4 tw=99:
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
/*
* This code implements an incremental mark-and-sweep garbage collector, with
* most sweeping carried out in the background on a parallel thread.
*
* Full vs. zone GC
* ----------------
*
* The collector can collect all zones at once, or a subset. These types of
* collection are referred to as a full GC and a zone GC respectively.
*
* The atoms zone is only collected in a full GC since objects in any zone may
* have pointers to atoms, and these are not recorded in the cross compartment
* pointer map. Also, the atoms zone is not collected if any thread has an
* AutoKeepAtoms instance on the stack, or there are any exclusive threads using
* the runtime.
*
* It is possible for an incremental collection that started out as a full GC to
* become a zone GC if new zones are created during the course of the
* collection.
*
* Incremental collection
* ----------------------
*
* For a collection to be carried out incrementally the following conditions
* must be met:
* - the collection must be run by calling js::GCSlice() rather than js::GC()
* - the GC mode must have been set to JSGC_MODE_INCREMENTAL with
* JS_SetGCParameter()
* - no thread may have an AutoKeepAtoms instance on the stack
* - all native objects that have their own trace hook must indicate that they
* implement read and write barriers with the JSCLASS_IMPLEMENTS_BARRIERS
* flag
*
* The last condition is an engine-internal mechanism to ensure that incremental
* collection is not carried out without the correct barriers being implemented.
* For more information see 'Incremental marking' below.
*
* If the collection is not incremental, all foreground activity happens inside
* a single call to GC() or GCSlice(). However the collection is not complete
* until the background sweeping activity has finished.
*
* An incremental collection proceeds as a series of slices, interleaved with
* mutator activity, i.e. running JavaScript code. Slices are limited by a time
* budget. The slice finishes as soon as possible after the requested time has
* passed.
*
* Collector states
* ----------------
*
* The collector proceeds through the following states, the current state being
* held in JSRuntime::gcIncrementalState:
*
* - MARK_ROOTS - marks the stack and other roots
* - MARK - incrementally marks reachable things
* - SWEEP - sweeps zones in groups and continues marking unswept zones
*
* The MARK_ROOTS activity always takes place in the first slice. The next two
* states can take place over one or more slices.
*
* In other words an incremental collection proceeds like this:
*
* Slice 1: MARK_ROOTS: Roots pushed onto the mark stack.
* MARK: The mark stack is processed by popping an element,
* marking it, and pushing its children.
*
* ... JS code runs ...
*
* Slice 2: MARK: More mark stack processing.
*
* ... JS code runs ...
*
* Slice n-1: MARK: More mark stack processing.
*
* ... JS code runs ...
*
* Slice n: MARK: Mark stack is completely drained.
* SWEEP: Select first group of zones to sweep and sweep them.
*
* ... JS code runs ...
*
* Slice n+1: SWEEP: Mark objects in unswept zones that were newly
* identified as alive (see below). Then sweep more zone
* groups.
*
* ... JS code runs ...
*
* Slice n+2: SWEEP: Mark objects in unswept zones that were newly
* identified as alive. Then sweep more zone groups.
*
* ... JS code runs ...
*
* Slice m: SWEEP: Sweeping is finished, and background sweeping
* started on the helper thread.
*
* ... JS code runs, remaining sweeping done on background thread ...
*
* When background sweeping finishes the GC is complete.
*
* Incremental marking
* -------------------
*
* Incremental collection requires close collaboration with the mutator (i.e.,
* JS code) to guarantee correctness.
*
* - During an incremental GC, if a memory location (except a root) is written
* to, then the value it previously held must be marked. Write barriers
* ensure this.
*
* - Any object that is allocated during incremental GC must start out marked.
*
* - Roots are marked in the first slice and hence don't need write barriers.
* Roots are things like the C stack and the VM stack.
*
* The problem that write barriers solve is that between slices the mutator can
* change the object graph. We must ensure that it cannot do this in such a way
* that makes us fail to mark a reachable object (marking an unreachable object
* is tolerable).
*
* We use a snapshot-at-the-beginning algorithm to do this. This means that we
* promise to mark at least everything that is reachable at the beginning of
* collection. To implement it we mark the old contents of every non-root memory
* location written to by the mutator while the collection is in progress, using
* write barriers. This is described in gc/Barrier.h.
*
* Incremental sweeping
* --------------------
*
* Sweeping is difficult to do incrementally because object finalizers must be
* run at the start of sweeping, before any mutator code runs. The reason is
* that some objects use their finalizers to remove themselves from caches. If
* mutator code was allowed to run after the start of sweeping, it could observe
* the state of the cache and create a new reference to an object that was just
* about to be destroyed.
*
* Sweeping all finalizable objects in one go would introduce long pauses, so
* instead sweeping broken up into groups of zones. Zones which are not yet
* being swept are still marked, so the issue above does not apply.
*
* The order of sweeping is restricted by cross compartment pointers - for
* example say that object |a| from zone A points to object |b| in zone B and
* neither object was marked when we transitioned to the SWEEP phase. Imagine we
* sweep B first and then return to the mutator. It's possible that the mutator
* could cause |a| to become alive through a read barrier (perhaps it was a
* shape that was accessed via a shape table). Then we would need to mark |b|,
* which |a| points to, but |b| has already been swept.
*
* So if there is such a pointer then marking of zone B must not finish before
* marking of zone A. Pointers which form a cycle between zones therefore
* restrict those zones to being swept at the same time, and these are found
* using Tarjan's algorithm for finding the strongly connected components of a
* graph.
*
* GC things without finalizers, and things with finalizers that are able to run
* in the background, are swept on the background thread. This accounts for most
* of the sweeping work.
*
* Reset
* -----
*
* During incremental collection it is possible, although unlikely, for
* conditions to change such that incremental collection is no longer safe. In
* this case, the collection is 'reset' by ResetIncrementalGC(). If we are in
* the mark state, this just stops marking, but if we have started sweeping
* already, we continue until we have swept the current zone group. Following a
* reset, a new non-incremental collection is started.
*/
#include "jsgcinlines.h"
#include "mozilla/ArrayUtils.h"
#include "mozilla/DebugOnly.h"
#include "mozilla/MemoryReporting.h"
#include "mozilla/Move.h"
#include <string.h> /* for memset used when DEBUG */
#ifndef XP_WIN
# include <unistd.h>
#endif
#include "jsapi.h"
#include "jsatom.h"
#include "jscntxt.h"
#include "jscompartment.h"
#include "jsobj.h"
#include "jsscript.h"
#include "jstypes.h"
#include "jsutil.h"
#include "jswatchpoint.h"
#include "jsweakmap.h"
#ifdef XP_WIN
# include "jswin.h"
#endif
#include "prmjtime.h"
#include "gc/FindSCCs.h"
#include "gc/GCInternals.h"
#include "gc/Marking.h"
#include "gc/Memory.h"
#ifdef JS_ION
# include "jit/BaselineJIT.h"
#endif
#include "jit/IonCode.h"
#include "js/SliceBudget.h"
#include "vm/Debugger.h"
#include "vm/ForkJoin.h"
#include "vm/ProxyObject.h"
#include "vm/Shape.h"
#include "vm/String.h"
#include "vm/TraceLogging.h"
#include "vm/WrapperObject.h"
#include "jsobjinlines.h"
#include "jsscriptinlines.h"
#include "vm/Stack-inl.h"
#include "vm/String-inl.h"
using namespace js;
using namespace js::gc;
using mozilla::ArrayEnd;
using mozilla::DebugOnly;
using mozilla::Maybe;
using mozilla::Swap;
/* Perform a Full GC every 20 seconds if MaybeGC is called */
static const uint64_t GC_IDLE_FULL_SPAN = 20 * 1000 * 1000;
/* Increase the IGC marking slice time if we are in highFrequencyGC mode. */
static const int IGC_MARK_SLICE_MULTIPLIER = 2;
#if defined(ANDROID) || defined(MOZ_B2G)
static const int MAX_EMPTY_CHUNK_COUNT = 2;
#else
static const int MAX_EMPTY_CHUNK_COUNT = 30;
#endif
/* This array should be const, but that doesn't link right under GCC. */
const AllocKind gc::slotsToThingKind[] = {
/* 0 */ FINALIZE_OBJECT0, FINALIZE_OBJECT2, FINALIZE_OBJECT2, FINALIZE_OBJECT4,
/* 4 */ FINALIZE_OBJECT4, FINALIZE_OBJECT8, FINALIZE_OBJECT8, FINALIZE_OBJECT8,
/* 8 */ FINALIZE_OBJECT8, FINALIZE_OBJECT12, FINALIZE_OBJECT12, FINALIZE_OBJECT12,
/* 12 */ FINALIZE_OBJECT12, FINALIZE_OBJECT16, FINALIZE_OBJECT16, FINALIZE_OBJECT16,
/* 16 */ FINALIZE_OBJECT16
};
static_assert(JS_ARRAY_LENGTH(slotsToThingKind) == SLOTS_TO_THING_KIND_LIMIT,
"We have defined a slot count for each kind.");
const uint32_t Arena::ThingSizes[] = {
sizeof(JSObject), /* FINALIZE_OBJECT0 */
sizeof(JSObject), /* FINALIZE_OBJECT0_BACKGROUND */
sizeof(JSObject_Slots2), /* FINALIZE_OBJECT2 */
sizeof(JSObject_Slots2), /* FINALIZE_OBJECT2_BACKGROUND */
sizeof(JSObject_Slots4), /* FINALIZE_OBJECT4 */
sizeof(JSObject_Slots4), /* FINALIZE_OBJECT4_BACKGROUND */
sizeof(JSObject_Slots8), /* FINALIZE_OBJECT8 */
sizeof(JSObject_Slots8), /* FINALIZE_OBJECT8_BACKGROUND */
sizeof(JSObject_Slots12), /* FINALIZE_OBJECT12 */
sizeof(JSObject_Slots12), /* FINALIZE_OBJECT12_BACKGROUND */
sizeof(JSObject_Slots16), /* FINALIZE_OBJECT16 */
sizeof(JSObject_Slots16), /* FINALIZE_OBJECT16_BACKGROUND */
sizeof(JSScript), /* FINALIZE_SCRIPT */
sizeof(LazyScript), /* FINALIZE_LAZY_SCRIPT */
sizeof(Shape), /* FINALIZE_SHAPE */
sizeof(BaseShape), /* FINALIZE_BASE_SHAPE */
sizeof(types::TypeObject), /* FINALIZE_TYPE_OBJECT */
sizeof(JSFatInlineString), /* FINALIZE_FAT_INLINE_STRING */
sizeof(JSString), /* FINALIZE_STRING */
sizeof(JSExternalString), /* FINALIZE_EXTERNAL_STRING */
sizeof(jit::JitCode), /* FINALIZE_JITCODE */
};
#define OFFSET(type) uint32_t(sizeof(ArenaHeader) + (ArenaSize - sizeof(ArenaHeader)) % sizeof(type))
const uint32_t Arena::FirstThingOffsets[] = {
OFFSET(JSObject), /* FINALIZE_OBJECT0 */
OFFSET(JSObject), /* FINALIZE_OBJECT0_BACKGROUND */
OFFSET(JSObject_Slots2), /* FINALIZE_OBJECT2 */
OFFSET(JSObject_Slots2), /* FINALIZE_OBJECT2_BACKGROUND */
OFFSET(JSObject_Slots4), /* FINALIZE_OBJECT4 */
OFFSET(JSObject_Slots4), /* FINALIZE_OBJECT4_BACKGROUND */
OFFSET(JSObject_Slots8), /* FINALIZE_OBJECT8 */
OFFSET(JSObject_Slots8), /* FINALIZE_OBJECT8_BACKGROUND */
OFFSET(JSObject_Slots12), /* FINALIZE_OBJECT12 */
OFFSET(JSObject_Slots12), /* FINALIZE_OBJECT12_BACKGROUND */
OFFSET(JSObject_Slots16), /* FINALIZE_OBJECT16 */
OFFSET(JSObject_Slots16), /* FINALIZE_OBJECT16_BACKGROUND */
OFFSET(JSScript), /* FINALIZE_SCRIPT */
OFFSET(LazyScript), /* FINALIZE_LAZY_SCRIPT */
OFFSET(Shape), /* FINALIZE_SHAPE */
OFFSET(BaseShape), /* FINALIZE_BASE_SHAPE */
OFFSET(types::TypeObject), /* FINALIZE_TYPE_OBJECT */
OFFSET(JSFatInlineString), /* FINALIZE_FAT_INLINE_STRING */
OFFSET(JSString), /* FINALIZE_STRING */
OFFSET(JSExternalString), /* FINALIZE_EXTERNAL_STRING */
OFFSET(jit::JitCode), /* FINALIZE_JITCODE */
};
#undef OFFSET
/*
* Finalization order for incrementally swept things.
*/
static const AllocKind FinalizePhaseStrings[] = {
FINALIZE_EXTERNAL_STRING
};
static const AllocKind FinalizePhaseScripts[] = {
FINALIZE_SCRIPT,
FINALIZE_LAZY_SCRIPT
};
static const AllocKind FinalizePhaseJitCode[] = {
FINALIZE_JITCODE
};
static const AllocKind * const FinalizePhases[] = {
FinalizePhaseStrings,
FinalizePhaseScripts,
FinalizePhaseJitCode
};
static const int FinalizePhaseCount = sizeof(FinalizePhases) / sizeof(AllocKind*);
static const int FinalizePhaseLength[] = {
sizeof(FinalizePhaseStrings) / sizeof(AllocKind),
sizeof(FinalizePhaseScripts) / sizeof(AllocKind),
sizeof(FinalizePhaseJitCode) / sizeof(AllocKind)
};
static const gcstats::Phase FinalizePhaseStatsPhase[] = {
gcstats::PHASE_SWEEP_STRING,
gcstats::PHASE_SWEEP_SCRIPT,
gcstats::PHASE_SWEEP_JITCODE
};
/*
* Finalization order for things swept in the background.
*/
static const AllocKind BackgroundPhaseObjects[] = {
FINALIZE_OBJECT0_BACKGROUND,
FINALIZE_OBJECT2_BACKGROUND,
FINALIZE_OBJECT4_BACKGROUND,
FINALIZE_OBJECT8_BACKGROUND,
FINALIZE_OBJECT12_BACKGROUND,
FINALIZE_OBJECT16_BACKGROUND
};
static const AllocKind BackgroundPhaseStrings[] = {
FINALIZE_FAT_INLINE_STRING,
FINALIZE_STRING
};
static const AllocKind BackgroundPhaseShapes[] = {
FINALIZE_SHAPE,
FINALIZE_BASE_SHAPE,
FINALIZE_TYPE_OBJECT
};
static const AllocKind * const BackgroundPhases[] = {
BackgroundPhaseObjects,
BackgroundPhaseStrings,
BackgroundPhaseShapes
};
static const int BackgroundPhaseCount = sizeof(BackgroundPhases) / sizeof(AllocKind*);
static const int BackgroundPhaseLength[] = {
sizeof(BackgroundPhaseObjects) / sizeof(AllocKind),
sizeof(BackgroundPhaseStrings) / sizeof(AllocKind),
sizeof(BackgroundPhaseShapes) / sizeof(AllocKind)
};
#ifdef DEBUG
void
ArenaHeader::checkSynchronizedWithFreeList() const
{
/*
* Do not allow to access the free list when its real head is still stored
* in FreeLists and is not synchronized with this one.
*/
JS_ASSERT(allocated());
/*
* We can be called from the background finalization thread when the free
* list in the zone can mutate at any moment. We cannot do any
* checks in this case.
*/
if (IsBackgroundFinalized(getAllocKind()) && zone->runtimeFromAnyThread()->gcHelperThread.onBackgroundThread())
return;
FreeSpan firstSpan = FreeSpan::decodeOffsets(arenaAddress(), firstFreeSpanOffsets);
if (firstSpan.isEmpty())
return;
const FreeSpan* list = zone->allocator.arenas.getFreeList(getAllocKind());
if (list->isEmpty() || firstSpan.arenaAddress() != list->arenaAddress())
return;
/*
* Here this arena has free things, FreeList::lists[thingKind] is not
* empty and also points to this arena. Thus they must the same.
*/
JS_ASSERT(firstSpan.isSameNonEmptySpan(list));
}
#endif
/* static */ void
Arena::staticAsserts()
{
static_assert(JS_ARRAY_LENGTH(ThingSizes) == FINALIZE_LIMIT, "We have defined all thing sizes.");
static_assert(JS_ARRAY_LENGTH(FirstThingOffsets) == FINALIZE_LIMIT, "We have defined all offsets.");
}
void
Arena::setAsFullyUnused(AllocKind thingKind)
{
FreeSpan entireList;
entireList.first = thingsStart(thingKind);
uintptr_t arenaAddr = aheader.arenaAddress();
entireList.last = arenaAddr | ArenaMask;
aheader.setFirstFreeSpan(&entireList);
}
template<typename T>
inline bool
Arena::finalize(FreeOp* fop, AllocKind thingKind, size_t thingSize)
{
/* Enforce requirements on size of T. */
JS_ASSERT(thingSize % CellSize == 0);
JS_ASSERT(thingSize <= 255);
JS_ASSERT(aheader.allocated());
JS_ASSERT(thingKind == aheader.getAllocKind());
JS_ASSERT(thingSize == aheader.getThingSize());
JS_ASSERT(!aheader.hasDelayedMarking);
JS_ASSERT(!aheader.markOverflow);
JS_ASSERT(!aheader.allocatedDuringIncremental);
uintptr_t thing = thingsStart(thingKind);
uintptr_t lastByte = thingsEnd() - 1;
FreeSpan nextFree(aheader.getFirstFreeSpan());
nextFree.checkSpan();
FreeSpan newListHead;
FreeSpan* newListTail = &newListHead;
uintptr_t newFreeSpanStart = 0;
bool allClear = true;
DebugOnly<size_t> nmarked = 0;
for (;; thing += thingSize) {
JS_ASSERT(thing <= lastByte + 1);
if (thing == nextFree.first) {
JS_ASSERT(nextFree.last <= lastByte);
if (nextFree.last == lastByte)
break;
JS_ASSERT(Arena::isAligned(nextFree.last, thingSize));
if (!newFreeSpanStart)
newFreeSpanStart = thing;
thing = nextFree.last;
nextFree = *nextFree.nextSpan();
nextFree.checkSpan();
} else {
T* t = reinterpret_cast<T*>(thing);
if (t->isMarked()) {
allClear = false;
nmarked++;
if (newFreeSpanStart) {
JS_ASSERT(thing >= thingsStart(thingKind) + thingSize);
newListTail->first = newFreeSpanStart;
newListTail->last = thing - thingSize;
newListTail = newListTail->nextSpanUnchecked(thingSize);
newFreeSpanStart = 0;
}
} else {
if (!newFreeSpanStart)
newFreeSpanStart = thing;
t->finalize(fop);
JS_POISON(t, JS_SWEPT_TENURED_PATTERN, thingSize);
}
}
}
if (allClear) {
JS_ASSERT(newListTail == &newListHead);
JS_ASSERT(!newFreeSpanStart ||
newFreeSpanStart == thingsStart(thingKind));
JS_EXTRA_POISON(data, JS_SWEPT_TENURED_PATTERN, sizeof(data));
return true;
}
newListTail->first = newFreeSpanStart ? newFreeSpanStart : nextFree.first;
JS_ASSERT(Arena::isAligned(newListTail->first, thingSize));
newListTail->last = lastByte;
#ifdef DEBUG
size_t nfree = 0;
for (const FreeSpan* span = &newListHead; span != newListTail; span = span->nextSpan()) {
span->checkSpan();
JS_ASSERT(Arena::isAligned(span->first, thingSize));
JS_ASSERT(Arena::isAligned(span->last, thingSize));
nfree += (span->last - span->first) / thingSize + 1;
JS_ASSERT(nfree + nmarked <= thingsPerArena(thingSize));
}
nfree += (newListTail->last + 1 - newListTail->first) / thingSize;
JS_ASSERT(nfree + nmarked == thingsPerArena(thingSize));
#endif
aheader.setFirstFreeSpan(&newListHead);
return false;
}
/*
* Insert an arena into the list in appropriate position and update the cursor
* to ensure that any arena before the cursor is full.
*/
void ArenaList::insert(ArenaHeader* a)
{
JS_ASSERT(a);
JS_ASSERT_IF(!head, cursor == &head);
a->next = *cursor;
*cursor = a;
if (!a->hasFreeThings())
cursor = &a->next;
}
template<typename T>
static inline bool
FinalizeTypedArenas(FreeOp* fop,
ArenaHeader** src,
ArenaList& dest,
AllocKind thingKind,
SliceBudget& budget)
{
/*
* Finalize arenas from src list, releasing empty arenas and inserting the
* others into dest in an appropriate position.
*/
/*
* During parallel sections, we sometimes finalize the parallel arenas,
* but in that case, we want to hold on to the memory in our arena
* lists, not offer it up for reuse.
*/
bool releaseArenas = !InParallelSection();
size_t thingSize = Arena::thingSize(thingKind);
while (ArenaHeader* aheader = *src) {
*src = aheader->next;
bool allClear = aheader->getArena()->finalize<T>(fop, thingKind, thingSize);
if (!allClear)
dest.insert(aheader);
else if (releaseArenas)
aheader->chunk()->releaseArena(aheader);
else
aheader->chunk()->recycleArena(aheader, dest, thingKind);
budget.step(Arena::thingsPerArena(thingSize));
if (budget.isOverBudget())
return false;
}
return true;
}
/*
* Finalize the list. On return al->cursor points to the first non-empty arena
* after the al->head.
*/
static bool
FinalizeArenas(FreeOp* fop,
ArenaHeader** src,
ArenaList& dest,
AllocKind thingKind,
SliceBudget& budget)
{
switch(thingKind) {
case FINALIZE_OBJECT0:
case FINALIZE_OBJECT0_BACKGROUND:
case FINALIZE_OBJECT2:
case FINALIZE_OBJECT2_BACKGROUND:
case FINALIZE_OBJECT4:
case FINALIZE_OBJECT4_BACKGROUND:
case FINALIZE_OBJECT8:
case FINALIZE_OBJECT8_BACKGROUND:
case FINALIZE_OBJECT12:
case FINALIZE_OBJECT12_BACKGROUND:
case FINALIZE_OBJECT16:
case FINALIZE_OBJECT16_BACKGROUND:
return FinalizeTypedArenas<JSObject>(fop, src, dest, thingKind, budget);
case FINALIZE_SCRIPT:
return FinalizeTypedArenas<JSScript>(fop, src, dest, thingKind, budget);
case FINALIZE_LAZY_SCRIPT:
return FinalizeTypedArenas<LazyScript>(fop, src, dest, thingKind, budget);
case FINALIZE_SHAPE:
return FinalizeTypedArenas<Shape>(fop, src, dest, thingKind, budget);
case FINALIZE_BASE_SHAPE:
return FinalizeTypedArenas<BaseShape>(fop, src, dest, thingKind, budget);
case FINALIZE_TYPE_OBJECT:
return FinalizeTypedArenas<types::TypeObject>(fop, src, dest, thingKind, budget);
case FINALIZE_STRING:
return FinalizeTypedArenas<JSString>(fop, src, dest, thingKind, budget);
case FINALIZE_FAT_INLINE_STRING:
return FinalizeTypedArenas<JSFatInlineString>(fop, src, dest, thingKind, budget);
case FINALIZE_EXTERNAL_STRING:
return FinalizeTypedArenas<JSExternalString>(fop, src, dest, thingKind, budget);
case FINALIZE_JITCODE:
#ifdef JS_ION
{
// JitCode finalization may release references on an executable
// allocator that is accessed when requesting interrupts.
JSRuntime::AutoLockForInterrupt lock(fop->runtime());
return FinalizeTypedArenas<jit::JitCode>(fop, src, dest, thingKind, budget);
}
#endif
default:
MOZ_ASSUME_UNREACHABLE("Invalid alloc kind");
}
}
static inline Chunk*
AllocChunk(JSRuntime* rt)
{
return static_cast<Chunk*>(MapAlignedPages(rt, ChunkSize, ChunkSize));
}
static inline void
FreeChunk(JSRuntime* rt, Chunk* p)
{
UnmapPages(rt, static_cast<void*>(p), ChunkSize);
}
inline bool
ChunkPool::wantBackgroundAllocation(JSRuntime* rt) const
{
/*
* To minimize memory waste we do not want to run the background chunk
* allocation if we have empty chunks or when the runtime needs just few
* of them.
*/
return rt->gcHelperThread.canBackgroundAllocate() &&
emptyCount == 0 &&
rt->gcChunkSet.count() >= 4;
}
/* Must be called with the GC lock taken. */
inline Chunk*
ChunkPool::get(JSRuntime* rt)
{
JS_ASSERT(this == &rt->gcChunkPool);
Chunk* chunk = emptyChunkListHead;
if (chunk) {
JS_ASSERT(emptyCount);
emptyChunkListHead = chunk->info.next;
--emptyCount;
} else {
JS_ASSERT(!emptyCount);
chunk = Chunk::allocate(rt);
if (!chunk)
return nullptr;
JS_ASSERT(chunk->info.numArenasFreeCommitted == 0);
}
JS_ASSERT(chunk->unused());
JS_ASSERT(!rt->gcChunkSet.has(chunk));
if (wantBackgroundAllocation(rt))
rt->gcHelperThread.startBackgroundAllocationIfIdle();
return chunk;
}
/* Must be called either during the GC or with the GC lock taken. */
inline void
ChunkPool::put(Chunk* chunk)
{
chunk->info.age = 0;
chunk->info.next = emptyChunkListHead;
emptyChunkListHead = chunk;
emptyCount++;
}
/* Must be called either during the GC or with the GC lock taken. */
Chunk*
ChunkPool::expire(JSRuntime* rt, bool releaseAll)
{
JS_ASSERT(this == &rt->gcChunkPool);
/*
* Return old empty chunks to the system while preserving the order of
* other chunks in the list. This way, if the GC runs several times
* without emptying the list, the older chunks will stay at the tail
* and are more likely to reach the max age.
*/
Chunk* freeList = nullptr;
int freeChunkCount = 0;
for (Chunk** chunkp = &emptyChunkListHead; *chunkp; ) {
JS_ASSERT(emptyCount);
Chunk* chunk = *chunkp;
JS_ASSERT(chunk->unused());
JS_ASSERT(!rt->gcChunkSet.has(chunk));
JS_ASSERT(chunk->info.age <= MAX_EMPTY_CHUNK_AGE);
if (releaseAll || chunk->info.age == MAX_EMPTY_CHUNK_AGE ||
freeChunkCount++ > MAX_EMPTY_CHUNK_COUNT)
{
*chunkp = chunk->info.next;
--emptyCount;
chunk->prepareToBeFreed(rt);
chunk->info.next = freeList;
freeList = chunk;
} else {
/* Keep the chunk but increase its age. */
++chunk->info.age;
chunkp = &chunk->info.next;
}
}
JS_ASSERT_IF(releaseAll, !emptyCount);
return freeList;
}
static void
FreeChunkList(JSRuntime* rt, Chunk* chunkListHead)
{
while (Chunk* chunk = chunkListHead) {
JS_ASSERT(!chunk->info.numArenasFreeCommitted);
chunkListHead = chunk->info.next;
FreeChunk(rt, chunk);
}
}
void
ChunkPool::expireAndFree(JSRuntime* rt, bool releaseAll)
{
FreeChunkList(rt, expire(rt, releaseAll));
}
/* static */ Chunk*
Chunk::allocate(JSRuntime* rt)
{
Chunk* chunk = AllocChunk(rt);
if (!chunk)
return nullptr;
chunk->init(rt);
rt->gcStats.count(gcstats::STAT_NEW_CHUNK);
return chunk;
}
/* Must be called with the GC lock taken. */
/* static */ inline void
Chunk::release(JSRuntime* rt, Chunk* chunk)
{
JS_ASSERT(chunk);
chunk->prepareToBeFreed(rt);
FreeChunk(rt, chunk);
}
inline void
Chunk::prepareToBeFreed(JSRuntime* rt)
{
JS_ASSERT(rt->gcNumArenasFreeCommitted >= info.numArenasFreeCommitted);
rt->gcNumArenasFreeCommitted -= info.numArenasFreeCommitted;
rt->gcStats.count(gcstats::STAT_DESTROY_CHUNK);
#ifdef DEBUG
/*
* Let FreeChunkList detect a missing prepareToBeFreed call before it
* frees chunk.
*/
info.numArenasFreeCommitted = 0;
#endif
}
void
Chunk::init(JSRuntime* rt)
{
JS_POISON(this, JS_FRESH_TENURED_PATTERN, ChunkSize);
/*
* We clear the bitmap to guard against xpc_IsGrayGCThing being called on
* uninitialized data, which would happen before the first GC cycle.
*/
bitmap.clear();
/*
* Decommit the arenas. We do this after poisoning so that if the OS does
* not have to recycle the pages, we still get the benefit of poisoning.
*/
decommitAllArenas(rt);
/* Initialize the chunk info. */
info.age = 0;
info.trailer.location = ChunkLocationTenuredHeap;
info.trailer.runtime = rt;
/* The rest of info fields are initialized in PickChunk. */
}
static inline Chunk**
GetAvailableChunkList(Zone* zone)
{
JSRuntime* rt = zone->runtimeFromAnyThread();
return zone->isSystem
? &rt->gcSystemAvailableChunkListHead
: &rt->gcUserAvailableChunkListHead;
}
inline void
Chunk::addToAvailableList(Zone* zone)
{
insertToAvailableList(GetAvailableChunkList(zone));
}
inline void
Chunk::insertToAvailableList(Chunk** insertPoint)
{
JS_ASSERT(hasAvailableArenas());
JS_ASSERT(!info.prevp);
JS_ASSERT(!info.next);
info.prevp = insertPoint;
Chunk* insertBefore = *insertPoint;
if (insertBefore) {
JS_ASSERT(insertBefore->info.prevp == insertPoint);
insertBefore->info.prevp = &info.next;
}
info.next = insertBefore;
*insertPoint = this;
}
inline void
Chunk::removeFromAvailableList()
{
JS_ASSERT(info.prevp);
*info.prevp = info.next;
if (info.next) {
JS_ASSERT(info.next->info.prevp == &info.next);
info.next->info.prevp = info.prevp;
}
info.prevp = nullptr;
info.next = nullptr;
}
/*
* Search for and return the next decommitted Arena. Our goal is to keep
* lastDecommittedArenaOffset "close" to a free arena. We do this by setting
* it to the most recently freed arena when we free, and forcing it to
* the last alloc + 1 when we allocate.
*/
uint32_t
Chunk::findDecommittedArenaOffset()
{
/* Note: lastFreeArenaOffset can be past the end of the list. */
for (unsigned i = info.lastDecommittedArenaOffset; i < ArenasPerChunk; i++)
if (decommittedArenas.get(i))
return i;
for (unsigned i = 0; i < info.lastDecommittedArenaOffset; i++)
if (decommittedArenas.get(i))
return i;
MOZ_ASSUME_UNREACHABLE("No decommitted arenas found.");
}
ArenaHeader*
Chunk::fetchNextDecommittedArena()
{
JS_ASSERT(info.numArenasFreeCommitted == 0);
JS_ASSERT(info.numArenasFree > 0);
unsigned offset = findDecommittedArenaOffset();
info.lastDecommittedArenaOffset = offset + 1;
--info.numArenasFree;
decommittedArenas.unset(offset);
Arena* arena = &arenas[offset];
MarkPagesInUse(info.trailer.runtime, arena, ArenaSize);
arena->aheader.setAsNotAllocated();
return &arena->aheader;
}
inline ArenaHeader*
Chunk::fetchNextFreeArena(JSRuntime* rt)
{
JS_ASSERT(info.numArenasFreeCommitted > 0);
JS_ASSERT(info.numArenasFreeCommitted <= info.numArenasFree);
JS_ASSERT(info.numArenasFreeCommitted <= rt->gcNumArenasFreeCommitted);
ArenaHeader* aheader = info.freeArenasHead;
info.freeArenasHead = aheader->next;
--info.numArenasFreeCommitted;
--info.numArenasFree;
--rt->gcNumArenasFreeCommitted;
return aheader;
}
ArenaHeader*
Chunk::allocateArena(Zone* zone, AllocKind thingKind)
{
JS_ASSERT(hasAvailableArenas());
JSRuntime* rt = zone->runtimeFromAnyThread();
if (!rt->isHeapMinorCollecting() && rt->gcBytes >= rt->gcMaxBytes)
return nullptr;
ArenaHeader* aheader = MOZ_LIKELY(info.numArenasFreeCommitted > 0)
? fetchNextFreeArena(rt)
: fetchNextDecommittedArena();
aheader->init(zone, thingKind);
if (MOZ_UNLIKELY(!hasAvailableArenas()))
removeFromAvailableList();
rt->gcBytes += ArenaSize;
zone->gcBytes += ArenaSize;
if (zone->gcBytes >= zone->gcTriggerBytes) {
AutoUnlockGC unlock(rt);
TriggerZoneGC(zone, JS::gcreason::ALLOC_TRIGGER);
}
return aheader;
}
inline void
Chunk::addArenaToFreeList(JSRuntime* rt, ArenaHeader* aheader)
{
JS_ASSERT(!aheader->allocated());
aheader->next = info.freeArenasHead;
info.freeArenasHead = aheader;
++info.numArenasFreeCommitted;
++info.numArenasFree;
++rt->gcNumArenasFreeCommitted;
}
void
Chunk::recycleArena(ArenaHeader* aheader, ArenaList& dest, AllocKind thingKind)
{
aheader->getArena()->setAsFullyUnused(thingKind);
dest.insert(aheader);
}
void
Chunk::releaseArena(ArenaHeader* aheader)
{
JS_ASSERT(aheader->allocated());
JS_ASSERT(!aheader->hasDelayedMarking);
Zone* zone = aheader->zone;
JSRuntime* rt = zone->runtimeFromAnyThread();
AutoLockGC maybeLock;
if (rt->gcHelperThread.sweeping())
maybeLock.lock(rt);
JS_ASSERT(rt->gcBytes >= ArenaSize);
JS_ASSERT(zone->gcBytes >= ArenaSize);
if (rt->gcHelperThread.sweeping())
zone->reduceGCTriggerBytes(zone->gcHeapGrowthFactor * ArenaSize);
rt->gcBytes -= ArenaSize;
zone->gcBytes -= ArenaSize;
aheader->setAsNotAllocated();
addArenaToFreeList(rt, aheader);
if (info.numArenasFree == 1) {
JS_ASSERT(!info.prevp);
JS_ASSERT(!info.next);
addToAvailableList(zone);
} else if (!unused()) {
JS_ASSERT(info.prevp);
} else {
rt->gcChunkSet.remove(this);
removeFromAvailableList();
JS_ASSERT(info.numArenasFree == ArenasPerChunk);
decommitAllArenas(rt);
rt->gcChunkPool.put(this);
}
}
/* The caller must hold the GC lock. */
static Chunk*
PickChunk(Zone* zone)
{
JSRuntime* rt = zone->runtimeFromAnyThread();
Chunk** listHeadp = GetAvailableChunkList(zone);
Chunk* chunk = *listHeadp;
if (chunk)
return chunk;
chunk = rt->gcChunkPool.get(rt);
if (!chunk)
return nullptr;
rt->gcChunkAllocationSinceLastGC = true;
/*
* FIXME bug 583732 - chunk is newly allocated and cannot be present in
* the table so using ordinary lookupForAdd is suboptimal here.
*/
GCChunkSet::AddPtr p = rt->gcChunkSet.lookupForAdd(chunk);
JS_ASSERT(!p);
if (!rt->gcChunkSet.add(p, chunk)) {
Chunk::release(rt, chunk);
return nullptr;
}
chunk->info.prevp = nullptr;
chunk->info.next = nullptr;
chunk->addToAvailableList(zone);
return chunk;
}
#ifdef JS_GC_ZEAL
extern void
js::SetGCZeal(JSRuntime* rt, uint8_t zeal, uint32_t frequency)
{
if (rt->gcVerifyPreData)
VerifyBarriers(rt, PreBarrierVerifier);
if (rt->gcVerifyPostData)
VerifyBarriers(rt, PostBarrierVerifier);
#ifdef JSGC_GENERATIONAL
if (rt->gcZeal_ == ZealGenerationalGCValue) {
MinorGC(rt, JS::gcreason::DEBUG_GC);
rt->gcNursery.leaveZealMode();
}
if (zeal == ZealGenerationalGCValue)
rt->gcNursery.enterZealMode();
#endif
bool schedule = zeal >= js::gc::ZealAllocValue;
rt->gcZeal_ = zeal;
rt->gcZealFrequency = frequency;
rt->gcNextScheduled = schedule ? frequency : 0;
}
static bool
InitGCZeal(JSRuntime* rt)
{
const char* env = getenv("JS_GC_ZEAL");
if (!env)
return true;
int zeal = -1;
int frequency = JS_DEFAULT_ZEAL_FREQ;
if (strcmp(env, "help") != 0) {
zeal = atoi(env);
const char* p = strchr(env, ',');
if (p)
frequency = atoi(p + 1);
}
if (zeal < 0 || zeal > ZealLimit || frequency < 0) {
fprintf(stderr,
"Format: JS_GC_ZEAL=N[,F]\n"
"N indicates \"zealousness\":\n"
" 0: no additional GCs\n"
" 1: additional GCs at common danger points\n"
" 2: GC every F allocations (default: 100)\n"
" 3: GC when the window paints (browser only)\n"
" 4: Verify pre write barriers between instructions\n"
" 5: Verify pre write barriers between paints\n"
" 6: Verify stack rooting\n"
" 7: Collect the nursery every N nursery allocations\n"
" 8: Incremental GC in two slices: 1) mark roots 2) finish collection\n"
" 9: Incremental GC in two slices: 1) mark all 2) new marking and finish\n"
" 10: Incremental GC in multiple slices\n"
" 11: Verify post write barriers between instructions\n"
" 12: Verify post write barriers between paints\n"
" 13: Purge analysis state every F allocations (default: 100)\n");
return false;
}
SetGCZeal(rt, zeal, frequency);
return true;
}
#endif
/* Lifetime for type sets attached to scripts containing observed types. */
static const int64_t JIT_SCRIPT_RELEASE_TYPES_INTERVAL = 60 * 1000 * 1000;
bool
js_InitGC(JSRuntime* rt, uint32_t maxbytes)
{
InitMemorySubsystem(rt);
if (!rt->gcChunkSet.init(INITIAL_CHUNK_CAPACITY))
return false;
if (!rt->gcRootsHash.init(256))
return false;
if (!rt->gcHelperThread.init())
return false;
/*
* Separate gcMaxMallocBytes from gcMaxBytes but initialize to maxbytes
* for default backward API compatibility.
*/
rt->gcMaxBytes = maxbytes;
rt->setGCMaxMallocBytes(maxbytes);
#ifndef JS_MORE_DETERMINISTIC
rt->gcJitReleaseTime = PRMJ_Now() + JIT_SCRIPT_RELEASE_TYPES_INTERVAL;
#endif
#ifdef JSGC_GENERATIONAL
if (!rt->gcNursery.init())
return false;
if (!rt->gcStoreBuffer.enable())
return false;
#endif
#ifdef JS_GC_ZEAL
if (!InitGCZeal(rt))
return false;
#endif
return true;
}
static void
RecordNativeStackTopForGC(JSRuntime* rt)
{
ConservativeGCData* cgcd = &rt->conservativeGC;
#ifdef JS_THREADSAFE
/* Record the stack top here only if we are called from a request. */
if (!rt->requestDepth)
return;
#endif
cgcd->recordStackTop();
}
void
js_FinishGC(JSRuntime* rt)
{
/*
* Wait until the background finalization stops and the helper thread
* shuts down before we forcefully release any remaining GC memory.
*/
rt->gcHelperThread.finish();
#ifdef JS_GC_ZEAL
/* Free memory associated with GC verification. */
FinishVerifier(rt);
#endif
/* Delete all remaining zones. */
if (rt->gcInitialized) {
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next())
js_delete(comp.get());
js_delete(zone.get());
}
}
rt->zones.clear();
rt->gcSystemAvailableChunkListHead = nullptr;
rt->gcUserAvailableChunkListHead = nullptr;
if (rt->gcChunkSet.initialized()) {
for (GCChunkSet::Range r(rt->gcChunkSet.all()); !r.empty(); r.popFront())
Chunk::release(rt, r.front());
rt->gcChunkSet.clear();
}
rt->gcChunkPool.expireAndFree(rt, true);
if (rt->gcRootsHash.initialized())
rt->gcRootsHash.clear();
rt->functionPersistentRooteds.clear();
rt->idPersistentRooteds.clear();
rt->objectPersistentRooteds.clear();
rt->scriptPersistentRooteds.clear();
rt->stringPersistentRooteds.clear();
rt->valuePersistentRooteds.clear();
}
template <typename T> struct BarrierOwner {};
template <typename T> struct BarrierOwner<T*> { typedef T result; };
template <> struct BarrierOwner<Value> { typedef HeapValue result; };
template <typename T>
static bool
AddRoot(JSRuntime* rt, T* rp, const char* name, JSGCRootType rootType)
{
/*
* Sometimes Firefox will hold weak references to objects and then convert
* them to strong references by calling AddRoot (e.g., via PreserveWrapper,
* or ModifyBusyCount in workers). We need a read barrier to cover these
* cases.
*/
if (rt->gcIncrementalState != NO_INCREMENTAL)
BarrierOwner<T>::result::writeBarrierPre(*rp);
return rt->gcRootsHash.put((void*)rp, RootInfo(name, rootType));
}
template <typename T>
static bool
AddRoot(JSContext* cx, T* rp, const char* name, JSGCRootType rootType)
{
bool ok = AddRoot(cx->runtime(), rp, name, rootType);
if (!ok)
JS_ReportOutOfMemory(cx);
return ok;
}
bool
js::AddValueRoot(JSContext* cx, Value* vp, const char* name)
{
return AddRoot(cx, vp, name, JS_GC_ROOT_VALUE_PTR);
}
extern bool
js::AddValueRootRT(JSRuntime* rt, js::Value* vp, const char* name)
{
return AddRoot(rt, vp, name, JS_GC_ROOT_VALUE_PTR);
}
extern bool
js::AddStringRoot(JSContext* cx, JSString** rp, const char* name)
{
return AddRoot(cx, rp, name, JS_GC_ROOT_STRING_PTR);
}
extern bool
js::AddObjectRoot(JSContext* cx, JSObject** rp, const char* name)
{
return AddRoot(cx, rp, name, JS_GC_ROOT_OBJECT_PTR);
}
extern bool
js::AddObjectRoot(JSRuntime* rt, JSObject** rp, const char* name)
{
return AddRoot(rt, rp, name, JS_GC_ROOT_OBJECT_PTR);
}
extern bool
js::AddScriptRoot(JSContext* cx, JSScript** rp, const char* name)
{
return AddRoot(cx, rp, name, JS_GC_ROOT_SCRIPT_PTR);
}
extern JS_FRIEND_API(bool)
js::AddRawValueRoot(JSContext* cx, Value* vp, const char* name)
{
return AddRoot(cx, vp, name, JS_GC_ROOT_VALUE_PTR);
}
extern JS_FRIEND_API(void)
js::RemoveRawValueRoot(JSContext* cx, Value* vp)
{
RemoveRoot(cx->runtime(), vp);
}
void
js::RemoveRoot(JSRuntime* rt, void* rp)
{
rt->gcRootsHash.remove(rp);
rt->gcPoke = true;
}
typedef RootedValueMap::Range RootRange;
typedef RootedValueMap::Entry RootEntry;
typedef RootedValueMap::Enum RootEnum;
static size_t
ComputeTriggerBytes(Zone* zone, size_t lastBytes, size_t maxBytes, JSGCInvocationKind gckind)
{
size_t base = gckind == GC_SHRINK ? lastBytes : Max(lastBytes, zone->runtimeFromMainThread()->gcAllocationThreshold);
double trigger = double(base) * zone->gcHeapGrowthFactor;
return size_t(Min(double(maxBytes), trigger));
}
void
Zone::setGCLastBytes(size_t lastBytes, JSGCInvocationKind gckind)
{
/*
* The heap growth factor depends on the heap size after a GC and the GC frequency.
* For low frequency GCs (more than 1sec between GCs) we let the heap grow to 150%.
* For high frequency GCs we let the heap grow depending on the heap size:
* lastBytes < highFrequencyLowLimit: 300%
* lastBytes > highFrequencyHighLimit: 150%
* otherwise: linear interpolation between 150% and 300% based on lastBytes
*/
JSRuntime* rt = runtimeFromMainThread();
if (!rt->gcDynamicHeapGrowth) {
gcHeapGrowthFactor = 3.0;
} else if (lastBytes < 1 * 1024 * 1024) {
gcHeapGrowthFactor = rt->gcLowFrequencyHeapGrowth;
} else {
JS_ASSERT(rt->gcHighFrequencyHighLimitBytes > rt->gcHighFrequencyLowLimitBytes);
uint64_t now = PRMJ_Now();
if (rt->gcLastGCTime && rt->gcLastGCTime + rt->gcHighFrequencyTimeThreshold * PRMJ_USEC_PER_MSEC > now) {
if (lastBytes <= rt->gcHighFrequencyLowLimitBytes) {
gcHeapGrowthFactor = rt->gcHighFrequencyHeapGrowthMax;
} else if (lastBytes >= rt->gcHighFrequencyHighLimitBytes) {
gcHeapGrowthFactor = rt->gcHighFrequencyHeapGrowthMin;
} else {
double k = (rt->gcHighFrequencyHeapGrowthMin - rt->gcHighFrequencyHeapGrowthMax)
/ (double)(rt->gcHighFrequencyHighLimitBytes - rt->gcHighFrequencyLowLimitBytes);
gcHeapGrowthFactor = (k * (lastBytes - rt->gcHighFrequencyLowLimitBytes)
+ rt->gcHighFrequencyHeapGrowthMax);
JS_ASSERT(gcHeapGrowthFactor <= rt->gcHighFrequencyHeapGrowthMax
&& gcHeapGrowthFactor >= rt->gcHighFrequencyHeapGrowthMin);
}
rt->gcHighFrequencyGC = true;
} else {
gcHeapGrowthFactor = rt->gcLowFrequencyHeapGrowth;
rt->gcHighFrequencyGC = false;
}
}
gcTriggerBytes = ComputeTriggerBytes(this, lastBytes, rt->gcMaxBytes, gckind);
}
void
Zone::reduceGCTriggerBytes(size_t amount)
{
JS_ASSERT(amount > 0);
JS_ASSERT(gcTriggerBytes >= amount);
if (gcTriggerBytes - amount < runtimeFromAnyThread()->gcAllocationThreshold * gcHeapGrowthFactor)
return;
gcTriggerBytes -= amount;
}
Allocator::Allocator(Zone* zone)
: zone_(zone)
{}
inline void
GCMarker::delayMarkingArena(ArenaHeader* aheader)
{
if (aheader->hasDelayedMarking) {
/* Arena already scheduled to be marked later */
return;
}
aheader->setNextDelayedMarking(unmarkedArenaStackTop);
unmarkedArenaStackTop = aheader;
markLaterArenas++;
}
void
GCMarker::delayMarkingChildren(const void* thing)
{
const Cell* cell = reinterpret_cast<const Cell*>(thing);
cell->arenaHeader()->markOverflow = 1;
delayMarkingArena(cell->arenaHeader());
}
inline void
ArenaLists::prepareForIncrementalGC(JSRuntime* rt)
{
for (size_t i = 0; i != FINALIZE_LIMIT; ++i) {
FreeSpan* headSpan = &freeLists[i];
if (!headSpan->isEmpty()) {
ArenaHeader* aheader = headSpan->arenaHeader();
aheader->allocatedDuringIncremental = true;
rt->gcMarker.delayMarkingArena(aheader);
}
}
}
static inline void
PushArenaAllocatedDuringSweep(JSRuntime* runtime, ArenaHeader* arena)
{
arena->setNextAllocDuringSweep(runtime->gcArenasAllocatedDuringSweep);
runtime->gcArenasAllocatedDuringSweep = arena;
}
inline void*
ArenaLists::allocateFromArenaInline(Zone* zone, AllocKind thingKind)
{
/*
* Parallel JS Note:
*
* This function can be called from parallel threads all of which
* are associated with the same compartment. In that case, each
* thread will have a distinct ArenaLists. Therefore, whenever we
* fall through to PickChunk() we must be sure that we are holding
* a lock.
*/
Chunk* chunk = nullptr;
ArenaList* al = &arenaLists[thingKind];
AutoLockGC maybeLock;
#ifdef JS_THREADSAFE
volatile uintptr_t* bfs = &backgroundFinalizeState[thingKind];
if (*bfs != BFS_DONE) {
/*
* We cannot search the arena list for free things while the
* background finalization runs and can modify head or cursor at any
* moment. So we always allocate a new arena in that case.
*/
maybeLock.lock(zone->runtimeFromAnyThread());
if (*bfs == BFS_RUN) {
JS_ASSERT(!*al->cursor);
chunk = PickChunk(zone);
if (!chunk) {
/*
* Let the caller to wait for the background allocation to
* finish and restart the allocation attempt.
*/
return nullptr;
}
} else if (*bfs == BFS_JUST_FINISHED) {
/* See comments before BackgroundFinalizeState definition. */
*bfs = BFS_DONE;
} else {
JS_ASSERT(*bfs == BFS_DONE);
}
}
#endif /* JS_THREADSAFE */
if (!chunk) {
if (ArenaHeader* aheader = *al->cursor) {
JS_ASSERT(aheader->hasFreeThings());
/*
* Normally, the empty arenas are returned to the chunk
* and should not present on the list. In parallel
* execution, however, we keep empty arenas in the arena
* list to avoid synchronizing on the chunk.
*/
JS_ASSERT(!aheader->isEmpty() || InParallelSection());
al->cursor = &aheader->next;
/*
* Move the free span stored in the arena to the free list and
* allocate from it.
*/
freeLists[thingKind] = aheader->getFirstFreeSpan();
aheader->setAsFullyUsed();
if (MOZ_UNLIKELY(zone->wasGCStarted())) {
if (zone->needsBarrier()) {
aheader->allocatedDuringIncremental = true;
zone->runtimeFromMainThread()->gcMarker.delayMarkingArena(aheader);
} else if (zone->isGCSweeping()) {
PushArenaAllocatedDuringSweep(zone->runtimeFromMainThread(), aheader);
}
}
return freeLists[thingKind].infallibleAllocate(Arena::thingSize(thingKind));
}
/* Make sure we hold the GC lock before we call PickChunk. */
if (!maybeLock.locked())
maybeLock.lock(zone->runtimeFromAnyThread());
chunk = PickChunk(zone);
if (!chunk)
return nullptr;
}
/*
* While we still hold the GC lock get an arena from some chunk, mark it
* as full as its single free span is moved to the free lits, and insert
* it to the list as a fully allocated arena.
*
* We add the arena before the the head, not after the tail pointed by the
* cursor, so after the GC the most recently added arena will be used first
* for allocations improving cache locality.
*/
JS_ASSERT(!*al->cursor);
ArenaHeader* aheader = chunk->allocateArena(zone, thingKind);
if (!aheader)
return nullptr;
if (MOZ_UNLIKELY(zone->wasGCStarted())) {
if (zone->needsBarrier()) {
aheader->allocatedDuringIncremental = true;
zone->runtimeFromMainThread()->gcMarker.delayMarkingArena(aheader);
} else if (zone->isGCSweeping()) {
PushArenaAllocatedDuringSweep(zone->runtimeFromMainThread(), aheader);
}
}
aheader->next = al->head;
if (!al->head) {
JS_ASSERT(al->cursor == &al->head);
al->cursor = &aheader->next;
}
al->head = aheader;
/* See comments before allocateFromNewArena about this assert. */
JS_ASSERT(!aheader->hasFreeThings());
uintptr_t arenaAddr = aheader->arenaAddress();
return freeLists[thingKind].allocateFromNewArena(arenaAddr,
Arena::firstThingOffset(thingKind),
Arena::thingSize(thingKind));
}
void*
ArenaLists::allocateFromArena(JS::Zone* zone, AllocKind thingKind)
{
return allocateFromArenaInline(zone, thingKind);
}
void
ArenaLists::wipeDuringParallelExecution(JSRuntime* rt)
{
JS_ASSERT(InParallelSection());
// First, check that we all objects we have allocated are eligible
// for background finalization. The idea is that we will free
// (below) ALL background finalizable objects, because we know (by
// the rules of parallel execution) they are not reachable except
// by other thread-local objects. However, if there were any
// object ineligible for background finalization, it might retain
// a reference to one of these background finalizable objects, and
// that'd be bad.
for (unsigned i = 0; i < FINALIZE_LAST; i++) {
AllocKind thingKind = AllocKind(i);
if (!IsBackgroundFinalized(thingKind) && arenaLists[thingKind].head)
return;
}
// Finalize all background finalizable objects immediately and
// return the (now empty) arenas back to arena list.
FreeOp fop(rt, false);
for (unsigned i = 0; i < FINALIZE_OBJECT_LAST; i++) {
AllocKind thingKind = AllocKind(i);
if (!IsBackgroundFinalized(thingKind))
continue;
if (arenaLists[i].head) {
purge(thingKind);
forceFinalizeNow(&fop, thingKind);
}
}
}
void
ArenaLists::finalizeNow(FreeOp* fop, AllocKind thingKind)
{
JS_ASSERT(!IsBackgroundFinalized(thingKind));
forceFinalizeNow(fop, thingKind);
}
void
ArenaLists::forceFinalizeNow(FreeOp* fop, AllocKind thingKind)
{
JS_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
ArenaHeader* arenas = arenaLists[thingKind].head;
arenaLists[thingKind].clear();
SliceBudget budget;
FinalizeArenas(fop, &arenas, arenaLists[thingKind], thingKind, budget);
JS_ASSERT(!arenas);
}
void
ArenaLists::queueForForegroundSweep(FreeOp* fop, AllocKind thingKind)
{
JS_ASSERT(!IsBackgroundFinalized(thingKind));
JS_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
JS_ASSERT(!arenaListsToSweep[thingKind]);
arenaListsToSweep[thingKind] = arenaLists[thingKind].head;
arenaLists[thingKind].clear();
}
inline void
ArenaLists::queueForBackgroundSweep(FreeOp* fop, AllocKind thingKind)
{
JS_ASSERT(IsBackgroundFinalized(thingKind));
#ifdef JS_THREADSAFE
JS_ASSERT(!fop->runtime()->gcHelperThread.sweeping());
#endif
ArenaList* al = &arenaLists[thingKind];
if (!al->head) {
JS_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
JS_ASSERT(al->cursor == &al->head);
return;
}
/*
* The state can be done, or just-finished if we have not allocated any GC
* things from the arena list after the previous background finalization.
*/
JS_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE ||
backgroundFinalizeState[thingKind] == BFS_JUST_FINISHED);
arenaListsToSweep[thingKind] = al->head;
al->clear();
backgroundFinalizeState[thingKind] = BFS_RUN;
}
/*static*/ void
ArenaLists::backgroundFinalize(FreeOp* fop, ArenaHeader* listHead, bool onBackgroundThread)
{
JS_ASSERT(listHead);
AllocKind thingKind = listHead->getAllocKind();
Zone* zone = listHead->zone;
ArenaList finalized;
SliceBudget budget;
FinalizeArenas(fop, &listHead, finalized, thingKind, budget);
JS_ASSERT(!listHead);
/*
* After we finish the finalization al->cursor must point to the end of
* the head list as we emptied the list before the background finalization
* and the allocation adds new arenas before the cursor.
*/
ArenaLists* lists = &zone->allocator.arenas;
ArenaList* al = &lists->arenaLists[thingKind];
AutoLockGC lock(fop->runtime());
JS_ASSERT(lists->backgroundFinalizeState[thingKind] == BFS_RUN);
JS_ASSERT(!*al->cursor);
if (finalized.head) {
*al->cursor = finalized.head;
if (finalized.cursor != &finalized.head)
al->cursor = finalized.cursor;
}
/*
* We must set the state to BFS_JUST_FINISHED if we are running on the
* background thread and we have touched arenaList list, even if we add to
* the list only fully allocated arenas without any free things. It ensures
* that the allocation thread takes the GC lock and all writes to the free
* list elements are propagated. As we always take the GC lock when
* allocating new arenas from the chunks we can set the state to BFS_DONE if
* we have released all finalized arenas back to their chunks.
*/
if (onBackgroundThread && finalized.head)
lists->backgroundFinalizeState[thingKind] = BFS_JUST_FINISHED;
else
lists->backgroundFinalizeState[thingKind] = BFS_DONE;
lists->arenaListsToSweep[thingKind] = nullptr;
}
void
ArenaLists::queueObjectsForSweep(FreeOp* fop)
{
gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_OBJECT);
finalizeNow(fop, FINALIZE_OBJECT0);
finalizeNow(fop, FINALIZE_OBJECT2);
finalizeNow(fop, FINALIZE_OBJECT4);
finalizeNow(fop, FINALIZE_OBJECT8);
finalizeNow(fop, FINALIZE_OBJECT12);
finalizeNow(fop, FINALIZE_OBJECT16);
queueForBackgroundSweep(fop, FINALIZE_OBJECT0_BACKGROUND);
queueForBackgroundSweep(fop, FINALIZE_OBJECT2_BACKGROUND);
queueForBackgroundSweep(fop, FINALIZE_OBJECT4_BACKGROUND);
queueForBackgroundSweep(fop, FINALIZE_OBJECT8_BACKGROUND);
queueForBackgroundSweep(fop, FINALIZE_OBJECT12_BACKGROUND);
queueForBackgroundSweep(fop, FINALIZE_OBJECT16_BACKGROUND);
}
void
ArenaLists::queueStringsForSweep(FreeOp* fop)
{
gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_STRING);
queueForBackgroundSweep(fop, FINALIZE_FAT_INLINE_STRING);
queueForBackgroundSweep(fop, FINALIZE_STRING);
queueForForegroundSweep(fop, FINALIZE_EXTERNAL_STRING);
}
void
ArenaLists::queueScriptsForSweep(FreeOp* fop)
{
gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_SCRIPT);
queueForForegroundSweep(fop, FINALIZE_SCRIPT);
queueForForegroundSweep(fop, FINALIZE_LAZY_SCRIPT);
}
void
ArenaLists::queueJitCodeForSweep(FreeOp* fop)
{
gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_JITCODE);
queueForForegroundSweep(fop, FINALIZE_JITCODE);
}
void
ArenaLists::queueShapesForSweep(FreeOp* fop)
{
gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_SHAPE);
queueForBackgroundSweep(fop, FINALIZE_SHAPE);
queueForBackgroundSweep(fop, FINALIZE_BASE_SHAPE);
queueForBackgroundSweep(fop, FINALIZE_TYPE_OBJECT);
}
static void*
RunLastDitchGC(JSContext* cx, JS::Zone* zone, AllocKind thingKind)
{
/*
* In parallel sections, we do not attempt to refill the free list
* and hence do not encounter last ditch GC.
*/
JS_ASSERT(!InParallelSection());
PrepareZoneForGC(zone);
JSRuntime* rt = cx->runtime();
/* The last ditch GC preserves all atoms. */
AutoKeepAtoms keepAtoms(cx->perThreadData);
GC(rt, GC_NORMAL, JS::gcreason::LAST_DITCH);
/*
* The JSGC_END callback can legitimately allocate new GC
* things and populate the free list. If that happens, just
* return that list head.
*/
size_t thingSize = Arena::thingSize(thingKind);
if (void* thing = zone->allocator.arenas.allocateFromFreeList(thingKind, thingSize))
return thing;
return nullptr;
}
template <AllowGC allowGC>
/* static */ void*
ArenaLists::refillFreeList(ThreadSafeContext* cx, AllocKind thingKind)
{
JS_ASSERT(cx->allocator()->arenas.freeLists[thingKind].isEmpty());
JS_ASSERT_IF(cx->isJSContext(), !cx->asJSContext()->runtime()->isHeapBusy());
Zone* zone = cx->allocator()->zone_;
bool runGC = cx->allowGC() && allowGC &&
cx->asJSContext()->runtime()->gcIncrementalState != NO_INCREMENTAL &&
zone->gcBytes > zone->gcTriggerBytes;
#ifdef JS_THREADSAFE
JS_ASSERT_IF(cx->isJSContext() && allowGC,
!cx->asJSContext()->runtime()->currentThreadHasExclusiveAccess());
#endif
for (;;) {
if (MOZ_UNLIKELY(runGC)) {
if (void* thing = RunLastDitchGC(cx->asJSContext(), zone, thingKind))
return thing;
}
if (cx->isJSContext()) {
/*
* allocateFromArena may fail while the background finalization still
* run. If we are on the main thread, we want to wait for it to finish
* and restart. However, checking for that is racy as the background
* finalization could free some things after allocateFromArena decided
* to fail but at this point it may have already stopped. To avoid
* this race we always try to allocate twice.
*/
for (bool secondAttempt = false; ; secondAttempt = true) {
void* thing = cx->allocator()->arenas.allocateFromArenaInline(zone, thingKind);
if (MOZ_LIKELY(!!thing))
return thing;
if (secondAttempt)
break;
cx->asJSContext()->runtime()->gcHelperThread.waitBackgroundSweepEnd();
}
} else {
#ifdef JS_THREADSAFE
/*
* If we're off the main thread, we try to allocate once and
* return whatever value we get. If we aren't in a ForkJoin
* session (i.e. we are in a worker thread async with the main
* thread), we need to first ensure the main thread is not in a GC
* session.
*/
mozilla::Maybe<AutoLockWorkerThreadState> lock;
JSRuntime* rt = zone->runtimeFromAnyThread();
if (rt->exclusiveThreadsPresent()) {
lock.construct();
while (rt->isHeapBusy())
WorkerThreadState().wait(GlobalWorkerThreadState::PRODUCER);
}
void* thing = cx->allocator()->arenas.allocateFromArenaInline(zone, thingKind);
if (thing)
return thing;
#else
MOZ_CRASH();
#endif
}
if (!cx->allowGC() || !allowGC)
return nullptr;
/*
* We failed to allocate. Run the GC if we haven't done it already.
* Otherwise report OOM.
*/
if (runGC)
break;
runGC = true;
}
JS_ASSERT(allowGC);
js_ReportOutOfMemory(cx);
return nullptr;
}
template void*
ArenaLists::refillFreeList<NoGC>(ThreadSafeContext* cx, AllocKind thingKind);
template void*
ArenaLists::refillFreeList<CanGC>(ThreadSafeContext* cx, AllocKind thingKind);
JSGCTraceKind
js_GetGCThingTraceKind(void* thing)
{
return GetGCThingTraceKind(thing);
}
/* static */ int64_t
SliceBudget::TimeBudget(int64_t millis)
{
return millis * PRMJ_USEC_PER_MSEC;
}
/* static */ int64_t
SliceBudget::WorkBudget(int64_t work)
{
/* For work = 0 not to mean Unlimited, we subtract 1. */
return -work - 1;
}
SliceBudget::SliceBudget()
: deadline(INT64_MAX),
counter(INTPTR_MAX)
{
}
SliceBudget::SliceBudget(int64_t budget)
{
if (budget == Unlimited) {
deadline = INT64_MAX;
counter = INTPTR_MAX;
} else if (budget > 0) {
deadline = PRMJ_Now() + budget;
counter = CounterReset;
} else {
deadline = 0;
counter = -budget - 1;
}
}
bool
SliceBudget::checkOverBudget()
{
bool over = PRMJ_Now() > deadline;
if (!over)
counter = CounterReset;
return over;
}
void
js::MarkCompartmentActive(InterpreterFrame* fp)
{
fp->script()->compartment()->zone()->active = true;
}
static void
RequestInterrupt(JSRuntime* rt, JS::gcreason::Reason reason)
{
if (rt->gcIsNeeded)
return;
rt->gcIsNeeded = true;
rt->gcTriggerReason = reason;
rt->requestInterrupt(JSRuntime::RequestInterruptMainThread);
}
bool
js::TriggerGC(JSRuntime* rt, JS::gcreason::Reason reason)
{
/* Wait till end of parallel section to trigger GC. */
if (InParallelSection()) {
ForkJoinContext::current()->requestGC(reason);
return true;
}
/* Don't trigger GCs when allocating under the interrupt callback lock. */
if (rt->currentThreadOwnsInterruptLock())
return false;
JS_ASSERT(CurrentThreadCanAccessRuntime(rt));
/* GC is already running. */
if (rt->isHeapCollecting())
return false;
JS::PrepareForFullGC(rt);
RequestInterrupt(rt, reason);
return true;
}
bool
js::TriggerZoneGC(Zone* zone, JS::gcreason::Reason reason)
{
/*
* If parallel threads are running, wait till they
* are stopped to trigger GC.
*/
if (InParallelSection()) {
ForkJoinContext::current()->requestZoneGC(zone, reason);
return true;
}
/* Zones in use by a thread with an exclusive context can't be collected. */
if (zone->usedByExclusiveThread)
return false;
JSRuntime* rt = zone->runtimeFromMainThread();
/* Don't trigger GCs when allocating under the interrupt callback lock. */
if (rt->currentThreadOwnsInterruptLock())
return false;
/* GC is already running. */
if (rt->isHeapCollecting())
return false;
if (rt->gcZeal() == ZealAllocValue) {
TriggerGC(rt, reason);
return true;
}
if (rt->isAtomsZone(zone)) {
/* We can't do a zone GC of the atoms compartment. */
TriggerGC(rt, reason);
return true;
}
PrepareZoneForGC(zone);
RequestInterrupt(rt, reason);
return true;
}
void
js::MaybeGC(JSContext* cx)
{
JSRuntime* rt = cx->runtime();
JS_ASSERT(CurrentThreadCanAccessRuntime(rt));
if (rt->gcZeal() == ZealAllocValue || rt->gcZeal() == ZealPokeValue) {
JS::PrepareForFullGC(rt);
GC(rt, GC_NORMAL, JS::gcreason::MAYBEGC);
return;
}
if (rt->gcIsNeeded) {
GCSlice(rt, GC_NORMAL, JS::gcreason::MAYBEGC);
return;
}
double factor = rt->gcHighFrequencyGC ? 0.85 : 0.9;
Zone* zone = cx->zone();
if (zone->gcBytes > 1024 * 1024 &&
zone->gcBytes >= factor * zone->gcTriggerBytes &&
rt->gcIncrementalState == NO_INCREMENTAL &&
!rt->gcHelperThread.sweeping())
{
PrepareZoneForGC(zone);
GCSlice(rt, GC_NORMAL, JS::gcreason::MAYBEGC);
return;
}
#ifndef JS_MORE_DETERMINISTIC
/*
* Access to the counters and, on 32 bit, setting gcNextFullGCTime below
* is not atomic and a race condition could trigger or suppress the GC. We
* tolerate this.
*/
int64_t now = PRMJ_Now();
if (rt->gcNextFullGCTime && rt->gcNextFullGCTime <= now) {
if (rt->gcChunkAllocationSinceLastGC ||
rt->gcNumArenasFreeCommitted > rt->gcDecommitThreshold)
{
JS::PrepareForFullGC(rt);
GCSlice(rt, GC_SHRINK, JS::gcreason::MAYBEGC);
} else {
rt->gcNextFullGCTime = now + GC_IDLE_FULL_SPAN;
}
}
#endif
}
static void
DecommitArenasFromAvailableList(JSRuntime* rt, Chunk** availableListHeadp)
{
Chunk* chunk = *availableListHeadp;
if (!chunk)
return;
/*
* Decommit is expensive so we avoid holding the GC lock while calling it.
*
* We decommit from the tail of the list to minimize interference with the
* main thread that may start to allocate things at this point.
*
* The arena that is been decommitted outside the GC lock must not be
* available for allocations either via the free list or via the
* decommittedArenas bitmap. For that we just fetch the arena from the
* free list before the decommit pretending as it was allocated. If this
* arena also is the single free arena in the chunk, then we must remove
* from the available list before we release the lock so the allocation
* thread would not see chunks with no free arenas on the available list.
*
* After we retake the lock, we mark the arena as free and decommitted if
* the decommit was successful. We must also add the chunk back to the
* available list if we removed it previously or when the main thread
* have allocated all remaining free arenas in the chunk.
*
* We also must make sure that the aheader is not accessed again after we
* decommit the arena.
*/
JS_ASSERT(chunk->info.prevp == availableListHeadp);
while (Chunk* next = chunk->info.next) {
JS_ASSERT(next->info.prevp == &chunk->info.next);
chunk = next;
}
for (;;) {
while (chunk->info.numArenasFreeCommitted != 0) {
ArenaHeader* aheader = chunk->fetchNextFreeArena(rt);
Chunk** savedPrevp = chunk->info.prevp;
if (!chunk->hasAvailableArenas())
chunk->removeFromAvailableList();
size_t arenaIndex = Chunk::arenaIndex(aheader->arenaAddress());
bool ok;
{
/*
* If the main thread waits for the decommit to finish, skip
* potentially expensive unlock/lock pair on the contested
* lock.
*/
Maybe<AutoUnlockGC> maybeUnlock;
if (!rt->isHeapBusy())
maybeUnlock.construct(rt);
ok = MarkPagesUnused(rt, aheader->getArena(), ArenaSize);
}
if (ok) {
++chunk->info.numArenasFree;
chunk->decommittedArenas.set(arenaIndex);
} else {
chunk->addArenaToFreeList(rt, aheader);
}
JS_ASSERT(chunk->hasAvailableArenas());
JS_ASSERT(!chunk->unused());
if (chunk->info.numArenasFree == 1) {
/*
* Put the chunk back to the available list either at the
* point where it was before to preserve the available list
* that we enumerate, or, when the allocation thread has fully
* used all the previous chunks, at the beginning of the
* available list.
*/
Chunk** insertPoint = savedPrevp;
if (savedPrevp != availableListHeadp) {
Chunk* prev = Chunk::fromPointerToNext(savedPrevp);
if (!prev->hasAvailableArenas())
insertPoint = availableListHeadp;
}
chunk->insertToAvailableList(insertPoint);
} else {
JS_ASSERT(chunk->info.prevp);
}
if (rt->gcChunkAllocationSinceLastGC || !ok) {
/*
* The allocator thread has started to get new chunks. We should stop
* to avoid decommitting arenas in just allocated chunks.
*/
return;
}
}
/*
* chunk->info.prevp becomes null when the allocator thread consumed
* all chunks from the available list.
*/
JS_ASSERT_IF(chunk->info.prevp, *chunk->info.prevp == chunk);
if (chunk->info.prevp == availableListHeadp || !chunk->info.prevp)
break;
/*
* prevp exists and is not the list head. It must point to the next
* field of the previous chunk.
*/
chunk = chunk->getPrevious();
}
}
static void
DecommitArenas(JSRuntime* rt)
{
DecommitArenasFromAvailableList(rt, &rt->gcSystemAvailableChunkListHead);
DecommitArenasFromAvailableList(rt, &rt->gcUserAvailableChunkListHead);
}
/* Must be called with the GC lock taken. */
static void
ExpireChunksAndArenas(JSRuntime* rt, bool shouldShrink)
{
if (Chunk* toFree = rt->gcChunkPool.expire(rt, shouldShrink)) {
AutoUnlockGC unlock(rt);
FreeChunkList(rt, toFree);
}
if (shouldShrink)
DecommitArenas(rt);
}
static void
SweepBackgroundThings(JSRuntime* rt, bool onBackgroundThread)
{
/*
* We must finalize in the correct order, see comments in
* finalizeObjects.
*/
FreeOp fop(rt, false);
for (int phase = 0 ; phase < BackgroundPhaseCount ; ++phase) {
for (Zone* zone = rt->gcSweepingZones; zone; zone = zone->gcNextGraphNode) {
for (int index = 0 ; index < BackgroundPhaseLength[phase] ; ++index) {
AllocKind kind = BackgroundPhases[phase][index];
ArenaHeader* arenas = zone->allocator.arenas.arenaListsToSweep[kind];
if (arenas)
ArenaLists::backgroundFinalize(&fop, arenas, onBackgroundThread);
}
}
}
rt->gcSweepingZones = nullptr;
}
#ifdef JS_THREADSAFE
static void
AssertBackgroundSweepingFinished(JSRuntime* rt)
{
JS_ASSERT(!rt->gcSweepingZones);
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
for (unsigned i = 0; i < FINALIZE_LIMIT; ++i) {
JS_ASSERT(!zone->allocator.arenas.arenaListsToSweep[i]);
JS_ASSERT(zone->allocator.arenas.doneBackgroundFinalize(AllocKind(i)));
}
}
}
unsigned
js::GetCPUCount()
{
static unsigned ncpus = 0;
if (ncpus == 0) {
# ifdef XP_WIN
SYSTEM_INFO sysinfo;
GetSystemInfo(&sysinfo);
ncpus = unsigned(sysinfo.dwNumberOfProcessors);
# else
long n = sysconf(_SC_NPROCESSORS_ONLN);
ncpus = (n > 0) ? unsigned(n) : 1;
# endif
}
return ncpus;
}
#endif /* JS_THREADSAFE */
bool
GCHelperThread::init()
{
if (!rt->useHelperThreads()) {
backgroundAllocation = false;
return true;
}
#ifdef JS_THREADSAFE
if (!(wakeup = PR_NewCondVar(rt->gcLock)))
return false;
if (!(done = PR_NewCondVar(rt->gcLock)))
return false;
thread = PR_CreateThread(PR_USER_THREAD, threadMain, this, PR_PRIORITY_NORMAL,
PR_GLOBAL_THREAD, PR_JOINABLE_THREAD, 0);
if (!thread)
return false;
backgroundAllocation = (GetCPUCount() >= 2);
#endif /* JS_THREADSAFE */
return true;
}
void
GCHelperThread::finish()
{
if (!rt->useHelperThreads() || !rt->gcLock) {
JS_ASSERT(state == IDLE);
return;
}
#ifdef JS_THREADSAFE
PRThread* join = nullptr;
{
AutoLockGC lock(rt);
if (thread && state != SHUTDOWN) {
/*
* We cannot be in the ALLOCATING or CANCEL_ALLOCATION states as
* the allocations should have been stopped during the last GC.
*/
JS_ASSERT(state == IDLE || state == SWEEPING);
if (state == IDLE)
PR_NotifyCondVar(wakeup);
state = SHUTDOWN;
join = thread;
}
}
if (join) {
/* PR_DestroyThread is not necessary. */
PR_JoinThread(join);
}
if (wakeup)
PR_DestroyCondVar(wakeup);
if (done)
PR_DestroyCondVar(done);
#endif /* JS_THREADSAFE */
}
#ifdef JS_THREADSAFE
#ifdef MOZ_NUWA_PROCESS
extern "C" {
MFBT_API bool IsNuwaProcess();
MFBT_API void NuwaMarkCurrentThread(void (*recreate)(void*), void* arg);
}
#endif
/* static */
void
GCHelperThread::threadMain(void* arg)
{
PR_SetCurrentThreadName("JS GC Helper");
#ifdef MOZ_NUWA_PROCESS
if (IsNuwaProcess && IsNuwaProcess()) {
JS_ASSERT(NuwaMarkCurrentThread != nullptr);
NuwaMarkCurrentThread(nullptr, nullptr);
}
#endif
static_cast<GCHelperThread*>(arg)->threadLoop();
}
void
GCHelperThread::wait(PRCondVar* which)
{
rt->gcLockOwner = nullptr;
PR_WaitCondVar(which, PR_INTERVAL_NO_TIMEOUT);
#ifdef DEBUG
rt->gcLockOwner = PR_GetCurrentThread();
#endif
}
void
GCHelperThread::threadLoop()
{
AutoLockGC lock(rt);
TraceLogger* logger = TraceLoggerForCurrentThread();
/*
* Even on the first iteration the state can be SHUTDOWN or SWEEPING if
* the stop request or the GC and the corresponding startBackgroundSweep call
* happen before this thread has a chance to run.
*/
for (;;) {
switch (state) {
case SHUTDOWN:
return;
case IDLE:
wait(wakeup);
break;
case SWEEPING: {
AutoTraceLog logSweeping(logger, TraceLogger::GCSweeping);
doSweep();
if (state == SWEEPING)
state = IDLE;
PR_NotifyAllCondVar(done);
break;
}
case ALLOCATING: {
AutoTraceLog logAllocating(logger, TraceLogger::GCAllocation);
do {
Chunk* chunk;
{
AutoUnlockGC unlock(rt);
chunk = Chunk::allocate(rt);
}
/* OOM stops the background allocation. */
if (!chunk)
break;
JS_ASSERT(chunk->info.numArenasFreeCommitted == 0);
rt->gcChunkPool.put(chunk);
} while (state == ALLOCATING && rt->gcChunkPool.wantBackgroundAllocation(rt));
if (state == ALLOCATING)
state = IDLE;
break;
}
case CANCEL_ALLOCATION:
state = IDLE;
PR_NotifyAllCondVar(done);
break;
}
}
}
#endif /* JS_THREADSAFE */
void
GCHelperThread::startBackgroundSweep(bool shouldShrink)
{
JS_ASSERT(rt->useHelperThreads());
#ifdef JS_THREADSAFE
AutoLockGC lock(rt);
JS_ASSERT(state == IDLE);
JS_ASSERT(!sweepFlag);
sweepFlag = true;
shrinkFlag = shouldShrink;
state = SWEEPING;
PR_NotifyCondVar(wakeup);
#endif /* JS_THREADSAFE */
}
/* Must be called with the GC lock taken. */
void
GCHelperThread::startBackgroundShrink()
{
JS_ASSERT(rt->useHelperThreads());
#ifdef JS_THREADSAFE
switch (state) {
case IDLE:
JS_ASSERT(!sweepFlag);
shrinkFlag = true;
state = SWEEPING;
PR_NotifyCondVar(wakeup);
break;
case SWEEPING:
shrinkFlag = true;
break;
case ALLOCATING:
case CANCEL_ALLOCATION:
/*
* If we have started background allocation there is nothing to
* shrink.
*/
break;
case SHUTDOWN:
MOZ_ASSUME_UNREACHABLE("No shrink on shutdown");
}
#endif /* JS_THREADSAFE */
}
void
GCHelperThread::waitBackgroundSweepEnd()
{
if (!rt->useHelperThreads()) {
JS_ASSERT(state == IDLE);
return;
}
#ifdef JS_THREADSAFE
AutoLockGC lock(rt);
while (state == SWEEPING)
wait(done);
if (rt->gcIncrementalState == NO_INCREMENTAL)
AssertBackgroundSweepingFinished(rt);
#endif /* JS_THREADSAFE */
}
void
GCHelperThread::waitBackgroundSweepOrAllocEnd()
{
if (!rt->useHelperThreads()) {
JS_ASSERT(state == IDLE);
return;
}
#ifdef JS_THREADSAFE
AutoLockGC lock(rt);
if (state == ALLOCATING)
state = CANCEL_ALLOCATION;
while (state == SWEEPING || state == CANCEL_ALLOCATION)
wait(done);
if (rt->gcIncrementalState == NO_INCREMENTAL)
AssertBackgroundSweepingFinished(rt);
#endif /* JS_THREADSAFE */
}
/* Must be called with the GC lock taken. */
inline void
GCHelperThread::startBackgroundAllocationIfIdle()
{
JS_ASSERT(rt->useHelperThreads());
#ifdef JS_THREADSAFE
if (state == IDLE) {
state = ALLOCATING;
PR_NotifyCondVar(wakeup);
}
#endif /* JS_THREADSAFE */
}
void
GCHelperThread::replenishAndFreeLater(void* ptr)
{
JS_ASSERT(freeCursor == freeCursorEnd);
do {
if (freeCursor && !freeVector.append(freeCursorEnd - FREE_ARRAY_LENGTH))
break;
freeCursor = (void**) js_malloc(FREE_ARRAY_SIZE);
if (!freeCursor) {
freeCursorEnd = nullptr;
break;
}
freeCursorEnd = freeCursor + FREE_ARRAY_LENGTH;
*freeCursor++ = ptr;
return;
} while (false);
js_free(ptr);
}
#ifdef JS_THREADSAFE
/* Must be called with the GC lock taken. */
void
GCHelperThread::doSweep()
{
if (sweepFlag) {
sweepFlag = false;
AutoUnlockGC unlock(rt);
SweepBackgroundThings(rt, true);
if (freeCursor) {
void** array = freeCursorEnd - FREE_ARRAY_LENGTH;
freeElementsAndArray(array, freeCursor);
freeCursor = freeCursorEnd = nullptr;
} else {
JS_ASSERT(!freeCursorEnd);
}
for (void*** iter = freeVector.begin(); iter != freeVector.end(); ++iter) {
void** array = *iter;
freeElementsAndArray(array, array + FREE_ARRAY_LENGTH);
}
freeVector.resize(0);
rt->freeLifoAlloc.freeAll();
}
bool shrinking = shrinkFlag;
ExpireChunksAndArenas(rt, shrinking);
/*
* The main thread may have called ShrinkGCBuffers while
* ExpireChunksAndArenas(rt, false) was running, so we recheck the flag
* afterwards.
*/
if (!shrinking && shrinkFlag) {
shrinkFlag = false;
ExpireChunksAndArenas(rt, true);
}
}
#endif /* JS_THREADSAFE */
bool
GCHelperThread::onBackgroundThread()
{
#ifdef JS_THREADSAFE
return PR_GetCurrentThread() == getThread();
#else
return false;
#endif
}
static bool
ReleaseObservedTypes(JSRuntime* rt)
{
bool releaseTypes = rt->gcZeal() != 0;
#ifndef JS_MORE_DETERMINISTIC
int64_t now = PRMJ_Now();
if (now >= rt->gcJitReleaseTime)
releaseTypes = true;
if (releaseTypes)
rt->gcJitReleaseTime = now + JIT_SCRIPT_RELEASE_TYPES_INTERVAL;
#endif
return releaseTypes;
}
/*
* It's simpler if we preserve the invariant that every zone has at least one
* compartment. If we know we're deleting the entire zone, then
* SweepCompartments is allowed to delete all compartments. In this case,
* |keepAtleastOne| is false. If some objects remain in the zone so that it
* cannot be deleted, then we set |keepAtleastOne| to true, which prohibits
* SweepCompartments from deleting every compartment. Instead, it preserves an
* arbitrary compartment in the zone.
*/
static void
SweepCompartments(FreeOp* fop, Zone* zone, bool keepAtleastOne, bool destroyingRuntime)
{
JSRuntime* rt = zone->runtimeFromMainThread();
JSDestroyCompartmentCallback callback = rt->destroyCompartmentCallback;
JSCompartment** read = zone->compartments.begin();
JSCompartment** end = zone->compartments.end();
JSCompartment** write = read;
bool foundOne = false;
while (read < end) {
JSCompartment* comp = *read++;
JS_ASSERT(!rt->isAtomsCompartment(comp));
/*
* Don't delete the last compartment if all the ones before it were
* deleted and keepAtleastOne is true.
*/
bool dontDelete = read == end && !foundOne && keepAtleastOne;
if ((!comp->marked && !dontDelete) || destroyingRuntime) {
if (callback)
callback(fop, comp);
if (comp->principals)
JS_DropPrincipals(rt, comp->principals);
js_delete(comp);
} else {
*write++ = comp;
foundOne = true;
}
}
zone->compartments.resize(write - zone->compartments.begin());
JS_ASSERT_IF(keepAtleastOne, !zone->compartments.empty());
}
static void
SweepZones(FreeOp* fop, bool destroyingRuntime)
{
JSRuntime* rt = fop->runtime();
MOZ_ASSERT_IF(destroyingRuntime, rt->numActiveZoneIters == 0);
if (rt->numActiveZoneIters)
return;
JSZoneCallback callback = rt->destroyZoneCallback;
/* Skip the atomsCompartment zone. */
Zone** read = rt->zones.begin() + 1;
Zone** end = rt->zones.end();
Zone** write = read;
JS_ASSERT(rt->zones.length() >= 1);
JS_ASSERT(rt->isAtomsZone(rt->zones[0]));
while (read < end) {
Zone* zone = *read++;
if (zone->wasGCStarted()) {
if ((zone->allocator.arenas.arenaListsAreEmpty() && !zone->hasMarkedCompartments()) ||
destroyingRuntime)
{
zone->allocator.arenas.checkEmptyFreeLists();
if (callback)
callback(zone);
SweepCompartments(fop, zone, false, destroyingRuntime);
JS_ASSERT(zone->compartments.empty());
fop->delete_(zone);
continue;
}
SweepCompartments(fop, zone, true, destroyingRuntime);
}
*write++ = zone;
}
rt->zones.resize(write - rt->zones.begin());
}
static void
PurgeRuntime(JSRuntime* rt)
{
for (GCCompartmentsIter comp(rt); !comp.done(); comp.next())
comp->purge();
rt->freeLifoAlloc.transferUnusedFrom(&rt->tempLifoAlloc);
rt->interpreterStack().purge(rt);
rt->gsnCache.purge();
rt->scopeCoordinateNameCache.purge();
rt->newObjectCache.purge();
rt->nativeIterCache.purge();
rt->sourceDataCache.purge();
rt->evalCache.clear();
if (!rt->hasActiveCompilations())
rt->parseMapPool().purgeAll();
}
static bool
ShouldPreserveJITCode(JSCompartment* comp, int64_t currentTime)
{
JSRuntime* rt = comp->runtimeFromMainThread();
if (rt->gcShouldCleanUpEverything)
return false;
if (rt->alwaysPreserveCode)
return true;
if (comp->lastAnimationTime + PRMJ_USEC_PER_SEC >= currentTime)
return true;
return false;
}
#ifdef DEBUG
class CompartmentCheckTracer : public JSTracer
{
public:
CompartmentCheckTracer(JSRuntime* rt, JSTraceCallback callback)
: JSTracer(rt, callback)
{}
Cell* src;
JSGCTraceKind srcKind;
Zone* zone;
JSCompartment* compartment;
};
static bool
InCrossCompartmentMap(JSObject* src, Cell* dst, JSGCTraceKind dstKind)
{
JSCompartment* srccomp = src->compartment();
if (dstKind == JSTRACE_OBJECT) {
Value key = ObjectValue(*static_cast<JSObject*>(dst));
if (WrapperMap::Ptr p = srccomp->lookupWrapper(key)) {
if (*p->value().unsafeGet() == ObjectValue(*src))
return true;
}
}
/*
* If the cross-compartment edge is caused by the debugger, then we don't
* know the right hashtable key, so we have to iterate.
*/
for (JSCompartment::WrapperEnum e(srccomp); !e.empty(); e.popFront()) {
if (e.front().key().wrapped == dst && ToMarkable(e.front().value()) == src)
return true;
}
return false;
}
static void
CheckCompartment(CompartmentCheckTracer* trc, JSCompartment* thingCompartment,
Cell* thing, JSGCTraceKind kind)
{
JS_ASSERT(thingCompartment == trc->compartment ||
trc->runtime()->isAtomsCompartment(thingCompartment) ||
(trc->srcKind == JSTRACE_OBJECT &&
InCrossCompartmentMap((JSObject*)trc->src, thing, kind)));
}
static JSCompartment*
CompartmentOfCell(Cell* thing, JSGCTraceKind kind)
{
if (kind == JSTRACE_OBJECT)
return static_cast<JSObject*>(thing)->compartment();
else if (kind == JSTRACE_SHAPE)
return static_cast<Shape*>(thing)->compartment();
else if (kind == JSTRACE_BASE_SHAPE)
return static_cast<BaseShape*>(thing)->compartment();
else if (kind == JSTRACE_SCRIPT)
return static_cast<JSScript*>(thing)->compartment();
else
return nullptr;
}
static void
CheckCompartmentCallback(JSTracer* trcArg, void** thingp, JSGCTraceKind kind)
{
CompartmentCheckTracer* trc = static_cast<CompartmentCheckTracer*>(trcArg);
Cell* thing = (Cell*)*thingp;
JSCompartment* comp = CompartmentOfCell(thing, kind);
if (comp && trc->compartment) {
CheckCompartment(trc, comp, thing, kind);
} else {
JS_ASSERT(thing->tenuredZone() == trc->zone ||
trc->runtime()->isAtomsZone(thing->tenuredZone()));
}
}
static void
CheckForCompartmentMismatches(JSRuntime* rt)
{
if (rt->gcDisableStrictProxyCheckingCount)
return;
CompartmentCheckTracer trc(rt, CheckCompartmentCallback);
for (ZonesIter zone(rt, SkipAtoms); !zone.done(); zone.next()) {
trc.zone = zone;
for (size_t thingKind = 0; thingKind < FINALIZE_LAST; thingKind++) {
for (CellIterUnderGC i(zone, AllocKind(thingKind)); !i.done(); i.next()) {
trc.src = i.getCell();
trc.srcKind = MapAllocToTraceKind(AllocKind(thingKind));
trc.compartment = CompartmentOfCell(trc.src, trc.srcKind);
JS_TraceChildren(&trc, trc.src, trc.srcKind);
}
}
}
}
#endif
static bool
BeginMarkPhase(JSRuntime* rt)
{
int64_t currentTime = PRMJ_Now();
#ifdef DEBUG
if (rt->gcFullCompartmentChecks)
CheckForCompartmentMismatches(rt);
#endif
rt->gcIsFull = true;
bool any = false;
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
/* Assert that zone state is as we expect */
JS_ASSERT(!zone->isCollecting());
JS_ASSERT(!zone->compartments.empty());
for (unsigned i = 0; i < FINALIZE_LIMIT; ++i)
JS_ASSERT(!zone->allocator.arenas.arenaListsToSweep[i]);
/* Set up which zones will be collected. */
if (zone->isGCScheduled()) {
if (!rt->isAtomsZone(zone)) {
any = true;
zone->setGCState(Zone::Mark);
}
} else {
rt->gcIsFull = false;
}
zone->scheduledForDestruction = false;
zone->maybeAlive = false;
zone->setPreservingCode(false);
}
for (CompartmentsIter c(rt, WithAtoms); !c.done(); c.next()) {
JS_ASSERT(c->gcLiveArrayBuffers.empty());
c->marked = false;
if (ShouldPreserveJITCode(c, currentTime))
c->zone()->setPreservingCode(true);
}
/*
* Atoms are not in the cross-compartment map. So if there are any
* zones that are not being collected, we are not allowed to collect
* atoms. Otherwise, the non-collected zones could contain pointers
* to atoms that we would miss.
*
* keepAtoms() will only change on the main thread, which we are currently
* on. If the value of keepAtoms() changes between GC slices, then we'll
* cancel the incremental GC. See IsIncrementalGCSafe.
*/
if (rt->gcIsFull && !rt->keepAtoms()) {
Zone* atomsZone = rt->atomsCompartment()->zone();
if (atomsZone->isGCScheduled()) {
JS_ASSERT(!atomsZone->isCollecting());
atomsZone->setGCState(Zone::Mark);
any = true;
}
}
/* Check that at least one zone is scheduled for collection. */
if (!any)
return false;
/*
* At the end of each incremental slice, we call prepareForIncrementalGC,
* which marks objects in all arenas that we're currently allocating
* into. This can cause leaks if unreachable objects are in these
* arenas. This purge call ensures that we only mark arenas that have had
* allocations after the incremental GC started.
*/
if (rt->gcIsIncremental) {
for (GCZonesIter zone(rt); !zone.done(); zone.next())
zone->allocator.arenas.purge();
}
rt->gcMarker.start();
JS_ASSERT(!rt->gcMarker.callback);
JS_ASSERT(IS_GC_MARKING_TRACER(&rt->gcMarker));
/* For non-incremental GC the following sweep discards the jit code. */
if (rt->gcIsIncremental) {
for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_MARK_DISCARD_CODE);
zone->discardJitCode(rt->defaultFreeOp());
}
}
GCMarker* gcmarker = &rt->gcMarker;
rt->gcStartNumber = rt->gcNumber;
/*
* We must purge the runtime at the beginning of an incremental GC. The
* danger if we purge later is that the snapshot invariant of incremental
* GC will be broken, as follows. If some object is reachable only through
* some cache (say the dtoaCache) then it will not be part of the snapshot.
* If we purge after root marking, then the mutator could obtain a pointer
* to the object and start using it. This object might never be marked, so
* a GC hazard would exist.
*/
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_PURGE);
PurgeRuntime(rt);
}
/*
* Mark phase.
*/
gcstats::AutoPhase ap1(rt->gcStats, gcstats::PHASE_MARK);
gcstats::AutoPhase ap2(rt->gcStats, gcstats::PHASE_MARK_ROOTS);
for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
/* Unmark everything in the zones being collected. */
zone->allocator.arenas.unmarkAll();
}
for (GCCompartmentsIter c(rt); !c.done(); c.next()) {
/* Reset weak map list for the compartments being collected. */
WeakMapBase::resetCompartmentWeakMapList(c);
}
if (rt->gcIsFull)
UnmarkScriptData(rt);
MarkRuntime(gcmarker);
if (rt->gcIsIncremental)
BufferGrayRoots(gcmarker);
/*
* This code ensures that if a zone is "dead", then it will be
* collected in this GC. A zone is considered dead if its maybeAlive
* flag is false. The maybeAlive flag is set if:
* (1) the zone has incoming cross-compartment edges, or
* (2) an object in the zone was marked during root marking, either
* as a black root or a gray root.
* If the maybeAlive is false, then we set the scheduledForDestruction flag.
* At any time later in the GC, if we try to mark an object whose
* zone is scheduled for destruction, we will assert.
* NOTE: Due to bug 811587, we only assert if gcManipulatingDeadCompartments
* is true (e.g., if we're doing a brain transplant).
*
* The purpose of this check is to ensure that a zone that we would
* normally destroy is not resurrected by a read barrier or an
* allocation. This might happen during a function like JS_TransplantObject,
* which iterates over all compartments, live or dead, and operates on their
* objects. See bug 803376 for details on this problem. To avoid the
* problem, we are very careful to avoid allocation and read barriers during
* JS_TransplantObject and the like. The code here ensures that we don't
* regress.
*
* Note that there are certain cases where allocations or read barriers in
* dead zone are difficult to avoid. We detect such cases (via the
* gcObjectsMarkedInDeadCompartment counter) and redo any ongoing GCs after
* the JS_TransplantObject function has finished. This ensures that the dead
* zones will be cleaned up. See AutoMarkInDeadZone and
* AutoMaybeTouchDeadZones for details.
*/
/* Set the maybeAlive flag based on cross-compartment edges. */
for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next()) {
for (JSCompartment::WrapperEnum e(c); !e.empty(); e.popFront()) {
Cell* dst = e.front().key().wrapped;
dst->tenuredZone()->maybeAlive = true;
}
}
/*
* For black roots, code in gc/Marking.cpp will already have set maybeAlive
* during MarkRuntime.
*/
for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
if (!zone->maybeAlive && !rt->isAtomsZone(zone))
zone->scheduledForDestruction = true;
}
rt->gcFoundBlackGrayEdges = false;
return true;
}
template <class CompartmentIterT>
static void
MarkWeakReferences(JSRuntime* rt, gcstats::Phase phase)
{
GCMarker* gcmarker = &rt->gcMarker;
JS_ASSERT(gcmarker->isDrained());
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_MARK);
gcstats::AutoPhase ap1(rt->gcStats, phase);
for (;;) {
bool markedAny = false;
for (CompartmentIterT c(rt); !c.done(); c.next()) {
markedAny |= WatchpointMap::markCompartmentIteratively(c, gcmarker);
markedAny |= WeakMapBase::markCompartmentIteratively(c, gcmarker);
}
markedAny |= Debugger::markAllIteratively(gcmarker);
if (!markedAny)
break;
SliceBudget budget;
gcmarker->drainMarkStack(budget);
}
JS_ASSERT(gcmarker->isDrained());
}
static void
MarkWeakReferencesInCurrentGroup(JSRuntime* rt, gcstats::Phase phase)
{
MarkWeakReferences<GCCompartmentGroupIter>(rt, phase);
}
template <class ZoneIterT, class CompartmentIterT>
static void
MarkGrayReferences(JSRuntime* rt)
{
GCMarker* gcmarker = &rt->gcMarker;
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_MARK);
gcstats::AutoPhase ap1(rt->gcStats, gcstats::PHASE_SWEEP_MARK_GRAY);
gcmarker->setMarkColorGray();
if (gcmarker->hasBufferedGrayRoots()) {
for (ZoneIterT zone(rt); !zone.done(); zone.next())
gcmarker->markBufferedGrayRoots(zone);
} else {
JS_ASSERT(!rt->gcIsIncremental);
if (JSTraceDataOp op = rt->gcGrayRootTracer.op)
(*op)(gcmarker, rt->gcGrayRootTracer.data);
}
SliceBudget budget;
gcmarker->drainMarkStack(budget);
}
MarkWeakReferences<CompartmentIterT>(rt, gcstats::PHASE_SWEEP_MARK_GRAY_WEAK);
JS_ASSERT(gcmarker->isDrained());
gcmarker->setMarkColorBlack();
}
static void
MarkGrayReferencesInCurrentGroup(JSRuntime* rt)
{
MarkGrayReferences<GCZoneGroupIter, GCCompartmentGroupIter>(rt);
}
#ifdef DEBUG
static void
MarkAllWeakReferences(JSRuntime* rt, gcstats::Phase phase)
{
MarkWeakReferences<GCCompartmentsIter>(rt, phase);
}
static void
MarkAllGrayReferences(JSRuntime* rt)
{
MarkGrayReferences<GCZonesIter, GCCompartmentsIter>(rt);
}
class js::gc::MarkingValidator
{
public:
MarkingValidator(JSRuntime* rt);
~MarkingValidator();
void nonIncrementalMark();
void validate();
private:
JSRuntime* runtime;
bool initialized;
typedef HashMap<Chunk*, ChunkBitmap*, GCChunkHasher, SystemAllocPolicy> BitmapMap;
BitmapMap map;
};
js::gc::MarkingValidator::MarkingValidator(JSRuntime* rt)
: runtime(rt),
initialized(false)
{}
js::gc::MarkingValidator::~MarkingValidator()
{
if (!map.initialized())
return;
for (BitmapMap::Range r(map.all()); !r.empty(); r.popFront())
js_delete(r.front().value());
}
void
js::gc::MarkingValidator::nonIncrementalMark()
{
/*
* Perform a non-incremental mark for all collecting zones and record
* the results for later comparison.
*
* Currently this does not validate gray marking.
*/
if (!map.init())
return;
GCMarker* gcmarker = &runtime->gcMarker;
/* Save existing mark bits. */
for (GCChunkSet::Range r(runtime->gcChunkSet.all()); !r.empty(); r.popFront()) {
ChunkBitmap* bitmap = &r.front()->bitmap;
ChunkBitmap* entry = js_new<ChunkBitmap>();
if (!entry)
return;
memcpy((void*)entry->bitmap, (void*)bitmap->bitmap, sizeof(bitmap->bitmap));
if (!map.putNew(r.front(), entry))
return;
}
/*
* Temporarily clear the lists of live weakmaps and array buffers for the
* compartments we are collecting.
*/
WeakMapVector weakmaps;
ArrayBufferVector arrayBuffers;
for (GCCompartmentsIter c(runtime); !c.done(); c.next()) {
if (!WeakMapBase::saveCompartmentWeakMapList(c, weakmaps) ||
!ArrayBufferObject::saveArrayBufferList(c, arrayBuffers))
{
return;
}
}
/*
* After this point, the function should run to completion, so we shouldn't
* do anything fallible.
*/
initialized = true;
for (GCCompartmentsIter c(runtime); !c.done(); c.next()) {
WeakMapBase::resetCompartmentWeakMapList(c);
ArrayBufferObject::resetArrayBufferList(c);
}
/* Re-do all the marking, but non-incrementally. */
js::gc::State state = runtime->gcIncrementalState;
runtime->gcIncrementalState = MARK_ROOTS;
JS_ASSERT(gcmarker->isDrained());
gcmarker->reset();
for (GCChunkSet::Range r(runtime->gcChunkSet.all()); !r.empty(); r.popFront())
r.front()->bitmap.clear();
{
gcstats::AutoPhase ap1(runtime->gcStats, gcstats::PHASE_MARK);
gcstats::AutoPhase ap2(runtime->gcStats, gcstats::PHASE_MARK_ROOTS);
MarkRuntime(gcmarker, true);
}
{
gcstats::AutoPhase ap1(runtime->gcStats, gcstats::PHASE_MARK);
SliceBudget budget;
runtime->gcIncrementalState = MARK;
runtime->gcMarker.drainMarkStack(budget);
}
runtime->gcIncrementalState = SWEEP;
{
gcstats::AutoPhase ap(runtime->gcStats, gcstats::PHASE_SWEEP);
MarkAllWeakReferences(runtime, gcstats::PHASE_SWEEP_MARK_WEAK);
/* Update zone state for gray marking. */
for (GCZonesIter zone(runtime); !zone.done(); zone.next()) {
JS_ASSERT(zone->isGCMarkingBlack());
zone->setGCState(Zone::MarkGray);
}
MarkAllGrayReferences(runtime);
/* Restore zone state. */
for (GCZonesIter zone(runtime); !zone.done(); zone.next()) {
JS_ASSERT(zone->isGCMarkingGray());
zone->setGCState(Zone::Mark);
}
}
/* Take a copy of the non-incremental mark state and restore the original. */
for (GCChunkSet::Range r(runtime->gcChunkSet.all()); !r.empty(); r.popFront()) {
Chunk* chunk = r.front();
ChunkBitmap* bitmap = &chunk->bitmap;
ChunkBitmap* entry = map.lookup(chunk)->value();
Swap(*entry, *bitmap);
}
for (GCCompartmentsIter c(runtime); !c.done(); c.next()) {
WeakMapBase::resetCompartmentWeakMapList(c);
ArrayBufferObject::resetArrayBufferList(c);
}
WeakMapBase::restoreCompartmentWeakMapLists(weakmaps);
ArrayBufferObject::restoreArrayBufferLists(arrayBuffers);
runtime->gcIncrementalState = state;
}
void
js::gc::MarkingValidator::validate()
{
/*
* Validates the incremental marking for a single compartment by comparing
* the mark bits to those previously recorded for a non-incremental mark.
*/
if (!initialized)
return;
for (GCChunkSet::Range r(runtime->gcChunkSet.all()); !r.empty(); r.popFront()) {
Chunk* chunk = r.front();
BitmapMap::Ptr ptr = map.lookup(chunk);
if (!ptr)
continue; /* Allocated after we did the non-incremental mark. */
ChunkBitmap* bitmap = ptr->value();
ChunkBitmap* incBitmap = &chunk->bitmap;
for (size_t i = 0; i < ArenasPerChunk; i++) {
if (chunk->decommittedArenas.get(i))
continue;
Arena* arena = &chunk->arenas[i];
if (!arena->aheader.allocated())
continue;
if (!arena->aheader.zone->isGCSweeping())
continue;
if (arena->aheader.allocatedDuringIncremental)
continue;
AllocKind kind = arena->aheader.getAllocKind();
uintptr_t thing = arena->thingsStart(kind);
uintptr_t end = arena->thingsEnd();
while (thing < end) {
Cell* cell = (Cell*)thing;
/*
* If a non-incremental GC wouldn't have collected a cell, then
* an incremental GC won't collect it.
*/
JS_ASSERT_IF(bitmap->isMarked(cell, BLACK), incBitmap->isMarked(cell, BLACK));
/*
* If the cycle collector isn't allowed to collect an object
* after a non-incremental GC has run, then it isn't allowed to
* collected it after an incremental GC.
*/
JS_ASSERT_IF(!bitmap->isMarked(cell, GRAY), !incBitmap->isMarked(cell, GRAY));
thing += Arena::thingSize(kind);
}
}
}
}
#endif
static void
ComputeNonIncrementalMarkingForValidation(JSRuntime* rt)
{
#ifdef DEBUG
JS_ASSERT(!rt->gcMarkingValidator);
if (rt->gcIsIncremental && rt->gcValidate)
rt->gcMarkingValidator = js_new<MarkingValidator>(rt);
if (rt->gcMarkingValidator)
rt->gcMarkingValidator->nonIncrementalMark();
#endif
}
static void
ValidateIncrementalMarking(JSRuntime* rt)
{
#ifdef DEBUG
if (rt->gcMarkingValidator)
rt->gcMarkingValidator->validate();
#endif
}
static void
FinishMarkingValidation(JSRuntime* rt)
{
#ifdef DEBUG
js_delete(rt->gcMarkingValidator);
rt->gcMarkingValidator = nullptr;
#endif
}
static void
AssertNeedsBarrierFlagsConsistent(JSRuntime* rt)
{
#ifdef DEBUG
bool anyNeedsBarrier = false;
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next())
anyNeedsBarrier |= zone->needsBarrier();
JS_ASSERT(rt->needsBarrier() == anyNeedsBarrier);
#endif
}
static void
DropStringWrappers(JSRuntime* rt)
{
/*
* String "wrappers" are dropped on GC because their presence would require
* us to sweep the wrappers in all compartments every time we sweep a
* compartment group.
*/
for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next()) {
for (JSCompartment::WrapperEnum e(c); !e.empty(); e.popFront()) {
if (e.front().key().kind == CrossCompartmentKey::StringWrapper)
e.removeFront();
}
}
}
/*
* Group zones that must be swept at the same time.
*
* If compartment A has an edge to an unmarked object in compartment B, then we
* must not sweep A in a later slice than we sweep B. That's because a write
* barrier in A that could lead to the unmarked object in B becoming
* marked. However, if we had already swept that object, we would be in trouble.
*
* If we consider these dependencies as a graph, then all the compartments in
* any strongly-connected component of this graph must be swept in the same
* slice.
*
* Tarjan's algorithm is used to calculate the components.
*/
void
JSCompartment::findOutgoingEdges(ComponentFinder<JS::Zone>& finder)
{
for (js::WrapperMap::Enum e(crossCompartmentWrappers); !e.empty(); e.popFront()) {
CrossCompartmentKey::Kind kind = e.front().key().kind;
JS_ASSERT(kind != CrossCompartmentKey::StringWrapper);
Cell* other = e.front().key().wrapped;
if (kind == CrossCompartmentKey::ObjectWrapper) {
/*
* Add edge to wrapped object compartment if wrapped object is not
* marked black to indicate that wrapper compartment not be swept
* after wrapped compartment.
*/
if (!other->isMarked(BLACK) || other->isMarked(GRAY)) {
JS::Zone* w = other->tenuredZone();
if (w->isGCMarking())
finder.addEdgeTo(w);
}
} else {
JS_ASSERT(kind == CrossCompartmentKey::DebuggerScript ||
kind == CrossCompartmentKey::DebuggerSource ||
kind == CrossCompartmentKey::DebuggerObject ||
kind == CrossCompartmentKey::DebuggerEnvironment);
/*
* Add edge for debugger object wrappers, to ensure (in conjuction
* with call to Debugger::findCompartmentEdges below) that debugger
* and debuggee objects are always swept in the same group.
*/
JS::Zone* w = other->tenuredZone();
if (w->isGCMarking())
finder.addEdgeTo(w);
}
}
Debugger::findCompartmentEdges(zone(), finder);
}
void
Zone::findOutgoingEdges(ComponentFinder<JS::Zone>& finder)
{
/*
* Any compartment may have a pointer to an atom in the atoms
* compartment, and these aren't in the cross compartment map.
*/
JSRuntime* rt = runtimeFromMainThread();
if (rt->atomsCompartment()->zone()->isGCMarking())
finder.addEdgeTo(rt->atomsCompartment()->zone());
for (CompartmentsInZoneIter comp(this); !comp.done(); comp.next())
comp->findOutgoingEdges(finder);
}
static void
FindZoneGroups(JSRuntime* rt)
{
ComponentFinder<Zone> finder(rt->mainThread.nativeStackLimit[StackForSystemCode]);
if (!rt->gcIsIncremental)
finder.useOneComponent();
for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
JS_ASSERT(zone->isGCMarking());
finder.addNode(zone);
}
rt->gcZoneGroups = finder.getResultsList();
rt->gcCurrentZoneGroup = rt->gcZoneGroups;
rt->gcZoneGroupIndex = 0;
JS_ASSERT_IF(!rt->gcIsIncremental, !rt->gcCurrentZoneGroup->nextGroup());
}
static void
ResetGrayList(JSCompartment* comp);
static void
GetNextZoneGroup(JSRuntime* rt)
{
rt->gcCurrentZoneGroup = rt->gcCurrentZoneGroup->nextGroup();
++rt->gcZoneGroupIndex;
if (!rt->gcCurrentZoneGroup) {
rt->gcAbortSweepAfterCurrentGroup = false;
return;
}
if (!rt->gcIsIncremental)
ComponentFinder<Zone>::mergeGroups(rt->gcCurrentZoneGroup);
if (rt->gcAbortSweepAfterCurrentGroup) {
JS_ASSERT(!rt->gcIsIncremental);
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
JS_ASSERT(!zone->gcNextGraphComponent);
JS_ASSERT(zone->isGCMarking());
zone->setNeedsBarrier(false, Zone::UpdateIon);
zone->setGCState(Zone::NoGC);
zone->gcGrayRoots.clearAndFree();
}
rt->setNeedsBarrier(false);
AssertNeedsBarrierFlagsConsistent(rt);
for (GCCompartmentGroupIter comp(rt); !comp.done(); comp.next()) {
ArrayBufferObject::resetArrayBufferList(comp);
ResetGrayList(comp);
}
rt->gcAbortSweepAfterCurrentGroup = false;
rt->gcCurrentZoneGroup = nullptr;
}
}
/*
* Gray marking:
*
* At the end of collection, anything reachable from a gray root that has not
* otherwise been marked black must be marked gray.
*
* This means that when marking things gray we must not allow marking to leave
* the current compartment group, as that could result in things being marked
* grey when they might subsequently be marked black. To achieve this, when we
* find a cross compartment pointer we don't mark the referent but add it to a
* singly-linked list of incoming gray pointers that is stored with each
* compartment.
*
* The list head is stored in JSCompartment::gcIncomingGrayPointers and contains
* cross compartment wrapper objects. The next pointer is stored in the second
* extra slot of the cross compartment wrapper.
*
* The list is created during gray marking when one of the
* MarkCrossCompartmentXXX functions is called for a pointer that leaves the
* current compartent group. This calls DelayCrossCompartmentGrayMarking to
* push the referring object onto the list.
*
* The list is traversed and then unlinked in
* MarkIncomingCrossCompartmentPointers.
*/
static bool
IsGrayListObject(JSObject* obj)
{
JS_ASSERT(obj);
return obj->is<CrossCompartmentWrapperObject>() && !IsDeadProxyObject(obj);
}
/* static */ unsigned
ProxyObject::grayLinkSlot(JSObject* obj)
{
JS_ASSERT(IsGrayListObject(obj));
return ProxyObject::EXTRA_SLOT + 1;
}
#ifdef DEBUG
static void
AssertNotOnGrayList(JSObject* obj)
{
JS_ASSERT_IF(IsGrayListObject(obj),
obj->getReservedSlot(ProxyObject::grayLinkSlot(obj)).isUndefined());
}
#endif
static JSObject*
CrossCompartmentPointerReferent(JSObject* obj)
{
JS_ASSERT(IsGrayListObject(obj));
return &obj->as<ProxyObject>().private_().toObject();
}
static JSObject*
NextIncomingCrossCompartmentPointer(JSObject* prev, bool unlink)
{
unsigned slot = ProxyObject::grayLinkSlot(prev);
JSObject* next = prev->getReservedSlot(slot).toObjectOrNull();
JS_ASSERT_IF(next, IsGrayListObject(next));
if (unlink)
prev->setSlot(slot, UndefinedValue());
return next;
}
void
js::DelayCrossCompartmentGrayMarking(JSObject* src)
{
JS_ASSERT(IsGrayListObject(src));
/* Called from MarkCrossCompartmentXXX functions. */
unsigned slot = ProxyObject::grayLinkSlot(src);
JSObject* dest = CrossCompartmentPointerReferent(src);
JSCompartment* comp = dest->compartment();
if (src->getReservedSlot(slot).isUndefined()) {
src->setCrossCompartmentSlot(slot, ObjectOrNullValue(comp->gcIncomingGrayPointers));
comp->gcIncomingGrayPointers = src;
} else {
JS_ASSERT(src->getReservedSlot(slot).isObjectOrNull());
}
#ifdef DEBUG
/*
* Assert that the object is in our list, also walking the list to check its
* integrity.
*/
JSObject* obj = comp->gcIncomingGrayPointers;
bool found = false;
while (obj) {
if (obj == src)
found = true;
obj = NextIncomingCrossCompartmentPointer(obj, false);
}
JS_ASSERT(found);
#endif
}
static void
MarkIncomingCrossCompartmentPointers(JSRuntime* rt, const uint32_t color)
{
JS_ASSERT(color == BLACK || color == GRAY);
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_MARK);
static const gcstats::Phase statsPhases[] = {
gcstats::PHASE_SWEEP_MARK_INCOMING_BLACK,
gcstats::PHASE_SWEEP_MARK_INCOMING_GRAY
};
gcstats::AutoPhase ap1(rt->gcStats, statsPhases[color]);
bool unlinkList = color == GRAY;
for (GCCompartmentGroupIter c(rt); !c.done(); c.next()) {
JS_ASSERT_IF(color == GRAY, c->zone()->isGCMarkingGray());
JS_ASSERT_IF(color == BLACK, c->zone()->isGCMarkingBlack());
JS_ASSERT_IF(c->gcIncomingGrayPointers, IsGrayListObject(c->gcIncomingGrayPointers));
for (JSObject* src = c->gcIncomingGrayPointers;
src;
src = NextIncomingCrossCompartmentPointer(src, unlinkList))
{
JSObject* dst = CrossCompartmentPointerReferent(src);
JS_ASSERT(dst->compartment() == c);
if (color == GRAY) {
if (IsObjectMarked(&src) && src->isMarked(GRAY))
MarkGCThingUnbarriered(&rt->gcMarker, (void**)&dst,
"cross-compartment gray pointer");
} else {
if (IsObjectMarked(&src) && !src->isMarked(GRAY))
MarkGCThingUnbarriered(&rt->gcMarker, (void**)&dst,
"cross-compartment black pointer");
}
}
if (unlinkList)
c->gcIncomingGrayPointers = nullptr;
}
SliceBudget budget;
rt->gcMarker.drainMarkStack(budget);
}
static bool
RemoveFromGrayList(JSObject* wrapper)
{
if (!IsGrayListObject(wrapper))
return false;
unsigned slot = ProxyObject::grayLinkSlot(wrapper);
if (wrapper->getReservedSlot(slot).isUndefined())
return false; /* Not on our list. */
JSObject* tail = wrapper->getReservedSlot(slot).toObjectOrNull();
wrapper->setReservedSlot(slot, UndefinedValue());
JSCompartment* comp = CrossCompartmentPointerReferent(wrapper)->compartment();
JSObject* obj = comp->gcIncomingGrayPointers;
if (obj == wrapper) {
comp->gcIncomingGrayPointers = tail;
return true;
}
while (obj) {
unsigned slot = ProxyObject::grayLinkSlot(obj);
JSObject* next = obj->getReservedSlot(slot).toObjectOrNull();
if (next == wrapper) {
obj->setCrossCompartmentSlot(slot, ObjectOrNullValue(tail));
return true;
}
obj = next;
}
MOZ_ASSUME_UNREACHABLE("object not found in gray link list");
}
static void
ResetGrayList(JSCompartment* comp)
{
JSObject* src = comp->gcIncomingGrayPointers;
while (src)
src = NextIncomingCrossCompartmentPointer(src, true);
comp->gcIncomingGrayPointers = nullptr;
}
void
js::NotifyGCNukeWrapper(JSObject* obj)
{
/*
* References to target of wrapper are being removed, we no longer have to
* remember to mark it.
*/
RemoveFromGrayList(obj);
}
enum {
JS_GC_SWAP_OBJECT_A_REMOVED = 1 << 0,
JS_GC_SWAP_OBJECT_B_REMOVED = 1 << 1
};
unsigned
js::NotifyGCPreSwap(JSObject* a, JSObject* b)
{
/*
* Two objects in the same compartment are about to have had their contents
* swapped. If either of them are in our gray pointer list, then we remove
* them from the lists, returning a bitset indicating what happened.
*/
return (RemoveFromGrayList(a) ? JS_GC_SWAP_OBJECT_A_REMOVED : 0) |
(RemoveFromGrayList(b) ? JS_GC_SWAP_OBJECT_B_REMOVED : 0);
}
void
js::NotifyGCPostSwap(JSObject* a, JSObject* b, unsigned removedFlags)
{
/*
* Two objects in the same compartment have had their contents swapped. If
* either of them were in our gray pointer list, we re-add them again.
*/
if (removedFlags & JS_GC_SWAP_OBJECT_A_REMOVED)
DelayCrossCompartmentGrayMarking(b);
if (removedFlags & JS_GC_SWAP_OBJECT_B_REMOVED)
DelayCrossCompartmentGrayMarking(a);
}
static void
EndMarkingZoneGroup(JSRuntime* rt)
{
/*
* Mark any incoming black pointers from previously swept compartments
* whose referents are not marked. This can occur when gray cells become
* black by the action of UnmarkGray.
*/
MarkIncomingCrossCompartmentPointers(rt, BLACK);
MarkWeakReferencesInCurrentGroup(rt, gcstats::PHASE_SWEEP_MARK_WEAK);
/*
* Change state of current group to MarkGray to restrict marking to this
* group. Note that there may be pointers to the atoms compartment, and
* these will be marked through, as they are not marked with
* MarkCrossCompartmentXXX.
*/
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
JS_ASSERT(zone->isGCMarkingBlack());
zone->setGCState(Zone::MarkGray);
}
/* Mark incoming gray pointers from previously swept compartments. */
rt->gcMarker.setMarkColorGray();
MarkIncomingCrossCompartmentPointers(rt, GRAY);
rt->gcMarker.setMarkColorBlack();
/* Mark gray roots and mark transitively inside the current compartment group. */
MarkGrayReferencesInCurrentGroup(rt);
/* Restore marking state. */
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
JS_ASSERT(zone->isGCMarkingGray());
zone->setGCState(Zone::Mark);
}
JS_ASSERT(rt->gcMarker.isDrained());
}
static void
BeginSweepingZoneGroup(JSRuntime* rt)
{
/*
* Begin sweeping the group of zones in gcCurrentZoneGroup,
* performing actions that must be done before yielding to caller.
*/
bool sweepingAtoms = false;
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
/* Set the GC state to sweeping. */
JS_ASSERT(zone->isGCMarking());
zone->setGCState(Zone::Sweep);
/* Purge the ArenaLists before sweeping. */
zone->allocator.arenas.purge();
if (rt->isAtomsZone(zone))
sweepingAtoms = true;
if (rt->sweepZoneCallback)
rt->sweepZoneCallback(zone);
}
ValidateIncrementalMarking(rt);
FreeOp fop(rt, rt->gcSweepOnBackgroundThread);
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_FINALIZE_START);
if (rt->gcFinalizeCallback)
rt->gcFinalizeCallback(&fop, JSFINALIZE_GROUP_START, !rt->gcIsFull /* unused */);
}
if (sweepingAtoms) {
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_ATOMS);
rt->sweepAtoms();
}
/* Prune out dead views from ArrayBuffer's view lists. */
for (GCCompartmentGroupIter c(rt); !c.done(); c.next())
ArrayBufferObject::sweep(c);
/* Collect watch points associated with unreachable objects. */
WatchpointMap::sweepAll(rt);
/* Detach unreachable debuggers and global objects from each other. */
Debugger::sweepAll(&fop);
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_COMPARTMENTS);
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_DISCARD_CODE);
zone->discardJitCode(&fop);
}
bool releaseTypes = ReleaseObservedTypes(rt);
for (GCCompartmentGroupIter c(rt); !c.done(); c.next()) {
gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
c->sweep(&fop, releaseTypes && !c->zone()->isPreservingCode());
}
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
// If there is an OOM while sweeping types, the type information
// will be deoptimized so that it is still correct (i.e.
// overapproximates the possible types in the zone), but the
// constraints might not have been triggered on the deoptimization
// or even copied over completely. In this case, destroy all JIT
// code and new script addendums in the zone, the only things whose
// correctness depends on the type constraints.
bool oom = false;
zone->sweep(&fop, releaseTypes && !zone->isPreservingCode(), &oom);
if (oom) {
zone->setPreservingCode(false);
zone->discardJitCode(&fop);
zone->types.clearAllNewScriptAddendumsOnOOM();
}
}
}
/*
* Queue all GC things in all zones for sweeping, either in the
* foreground or on the background thread.
*
* Note that order is important here for the background case.
*
* Objects are finalized immediately but this may change in the future.
*/
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
zone->allocator.arenas.queueObjectsForSweep(&fop);
}
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
zone->allocator.arenas.queueStringsForSweep(&fop);
}
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
zone->allocator.arenas.queueScriptsForSweep(&fop);
}
#ifdef JS_ION
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
zone->allocator.arenas.queueJitCodeForSweep(&fop);
}
#endif
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
zone->allocator.arenas.queueShapesForSweep(&fop);
zone->allocator.arenas.gcShapeArenasToSweep =
zone->allocator.arenas.arenaListsToSweep[FINALIZE_SHAPE];
}
rt->gcSweepPhase = 0;
rt->gcSweepZone = rt->gcCurrentZoneGroup;
rt->gcSweepKindIndex = 0;
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_FINALIZE_END);
if (rt->gcFinalizeCallback)
rt->gcFinalizeCallback(&fop, JSFINALIZE_GROUP_END, !rt->gcIsFull /* unused */);
}
}
static void
EndSweepingZoneGroup(JSRuntime* rt)
{
/* Update the GC state for zones we have swept and unlink the list. */
for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
JS_ASSERT(zone->isGCSweeping());
zone->setGCState(Zone::Finished);
}
/* Reset the list of arenas marked as being allocated during sweep phase. */
while (ArenaHeader* arena = rt->gcArenasAllocatedDuringSweep) {
rt->gcArenasAllocatedDuringSweep = arena->getNextAllocDuringSweep();
arena->unsetAllocDuringSweep();
}
}
static void
BeginSweepPhase(JSRuntime* rt, bool destroyingRuntime)
{
/*
* Sweep phase.
*
* Finalize as we sweep, outside of rt->gcLock but with rt->isHeapBusy()
* true so that any attempt to allocate a GC-thing from a finalizer will
* fail, rather than nest badly and leave the unmarked newborn to be swept.
*/
JS_ASSERT(!rt->gcAbortSweepAfterCurrentGroup);
ComputeNonIncrementalMarkingForValidation(rt);
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP);
#ifdef JS_THREADSAFE
rt->gcSweepOnBackgroundThread = !destroyingRuntime && rt->useHelperThreads();
#endif
#ifdef DEBUG
for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next()) {
JS_ASSERT(!c->gcIncomingGrayPointers);
for (JSCompartment::WrapperEnum e(c); !e.empty(); e.popFront()) {
if (e.front().key().kind != CrossCompartmentKey::StringWrapper)
AssertNotOnGrayList(&e.front().value().get().toObject());
}
}
#endif
DropStringWrappers(rt);
FindZoneGroups(rt);
EndMarkingZoneGroup(rt);
BeginSweepingZoneGroup(rt);
}
bool
ArenaLists::foregroundFinalize(FreeOp* fop, AllocKind thingKind, SliceBudget& sliceBudget)
{
if (!arenaListsToSweep[thingKind])
return true;
ArenaList& dest = arenaLists[thingKind];
return FinalizeArenas(fop, &arenaListsToSweep[thingKind], dest, thingKind, sliceBudget);
}
static bool
DrainMarkStack(JSRuntime* rt, SliceBudget& sliceBudget, gcstats::Phase phase)
{
/* Run a marking slice and return whether the stack is now empty. */
gcstats::AutoPhase ap(rt->gcStats, phase);
return rt->gcMarker.drainMarkStack(sliceBudget);
}
static bool
SweepPhase(JSRuntime* rt, SliceBudget& sliceBudget)
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP);
FreeOp fop(rt, rt->gcSweepOnBackgroundThread);
bool finished = DrainMarkStack(rt, sliceBudget, gcstats::PHASE_SWEEP_MARK);
if (!finished)
return false;
for (;;) {
/* Finalize foreground finalized things. */
for (; rt->gcSweepPhase < FinalizePhaseCount ; ++rt->gcSweepPhase) {
gcstats::AutoPhase ap(rt->gcStats, FinalizePhaseStatsPhase[rt->gcSweepPhase]);
for (; rt->gcSweepZone; rt->gcSweepZone = rt->gcSweepZone->nextNodeInGroup()) {
Zone* zone = rt->gcSweepZone;
while (rt->gcSweepKindIndex < FinalizePhaseLength[rt->gcSweepPhase]) {
AllocKind kind = FinalizePhases[rt->gcSweepPhase][rt->gcSweepKindIndex];
if (!zone->allocator.arenas.foregroundFinalize(&fop, kind, sliceBudget))
return false; /* Yield to the mutator. */
++rt->gcSweepKindIndex;
}
rt->gcSweepKindIndex = 0;
}
rt->gcSweepZone = rt->gcCurrentZoneGroup;
}
/* Remove dead shapes from the shape tree, but don't finalize them yet. */
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_SHAPE);
for (; rt->gcSweepZone; rt->gcSweepZone = rt->gcSweepZone->nextNodeInGroup()) {
Zone* zone = rt->gcSweepZone;
while (ArenaHeader* arena = zone->allocator.arenas.gcShapeArenasToSweep) {
for (CellIterUnderGC i(arena); !i.done(); i.next()) {
Shape* shape = i.get<Shape>();
if (!shape->isMarked())
shape->sweep();
}
zone->allocator.arenas.gcShapeArenasToSweep = arena->next;
sliceBudget.step(Arena::thingsPerArena(Arena::thingSize(FINALIZE_SHAPE)));
if (sliceBudget.isOverBudget())
return false; /* Yield to the mutator. */
}
}
}
EndSweepingZoneGroup(rt);
GetNextZoneGroup(rt);
if (!rt->gcCurrentZoneGroup)
return true; /* We're finished. */
EndMarkingZoneGroup(rt);
BeginSweepingZoneGroup(rt);
}
}
static void
EndSweepPhase(JSRuntime* rt, JSGCInvocationKind gckind, bool destroyingRuntime)
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP);
FreeOp fop(rt, rt->gcSweepOnBackgroundThread);
JS_ASSERT_IF(destroyingRuntime, !rt->gcSweepOnBackgroundThread);
JS_ASSERT(rt->gcMarker.isDrained());
rt->gcMarker.stop();
/*
* Recalculate whether GC was full or not as this may have changed due to
* newly created zones. Can only change from full to not full.
*/
if (rt->gcIsFull) {
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
if (!zone->isCollecting()) {
rt->gcIsFull = false;
break;
}
}
}
/*
* If we found any black->gray edges during marking, we completely clear the
* mark bits of all uncollected zones, or if a reset has occured, zones that
* will no longer be collected. This is safe, although it may
* prevent the cycle collector from collecting some dead objects.
*/
if (rt->gcFoundBlackGrayEdges) {
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
if (!zone->isCollecting())
zone->allocator.arenas.unmarkAll();
}
}
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_DESTROY);
/*
* Sweep script filenames after sweeping functions in the generic loop
* above. In this way when a scripted function's finalizer destroys the
* script and calls rt->destroyScriptHook, the hook can still access the
* script's filename. See bug 323267.
*/
if (rt->gcIsFull)
SweepScriptData(rt);
/* Clear out any small pools that we're hanging on to. */
if (JSC::ExecutableAllocator* execAlloc = rt->maybeExecAlloc())
execAlloc->purge();
/*
* This removes compartments from rt->compartment, so we do it last to make
* sure we don't miss sweeping any compartments.
*/
if (!destroyingRuntime)
SweepZones(&fop, destroyingRuntime);
if (!rt->gcSweepOnBackgroundThread) {
/*
* Destroy arenas after we finished the sweeping so finalizers can
* safely use IsAboutToBeFinalized(). This is done on the
* GCHelperThread if possible. We acquire the lock only because
* Expire needs to unlock it for other callers.
*/
AutoLockGC lock(rt);
ExpireChunksAndArenas(rt, gckind == GC_SHRINK);
}
}
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_FINALIZE_END);
if (rt->gcFinalizeCallback)
rt->gcFinalizeCallback(&fop, JSFINALIZE_COLLECTION_END, !rt->gcIsFull);
/* If we finished a full GC, then the gray bits are correct. */
if (rt->gcIsFull)
rt->gcGrayBitsValid = true;
}
/* Set up list of zones for sweeping of background things. */
JS_ASSERT(!rt->gcSweepingZones);
for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
zone->gcNextGraphNode = rt->gcSweepingZones;
rt->gcSweepingZones = zone;
}
/* If not sweeping on background thread then we must do it here. */
if (!rt->gcSweepOnBackgroundThread) {
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_DESTROY);
SweepBackgroundThings(rt, false);
rt->freeLifoAlloc.freeAll();
/* Ensure the compartments get swept if it's the last GC. */
if (destroyingRuntime)
SweepZones(&fop, destroyingRuntime);
}
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
zone->setGCLastBytes(zone->gcBytes, gckind);
if (zone->isCollecting()) {
JS_ASSERT(zone->isGCFinished());
zone->setGCState(Zone::NoGC);
}
#ifdef DEBUG
JS_ASSERT(!zone->isCollecting());
JS_ASSERT(!zone->wasGCStarted());
for (unsigned i = 0 ; i < FINALIZE_LIMIT ; ++i) {
JS_ASSERT_IF(!IsBackgroundFinalized(AllocKind(i)) ||
!rt->gcSweepOnBackgroundThread,
!zone->allocator.arenas.arenaListsToSweep[i]);
}
#endif
}
#ifdef DEBUG
for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next()) {
JS_ASSERT(!c->gcIncomingGrayPointers);
JS_ASSERT(c->gcLiveArrayBuffers.empty());
for (JSCompartment::WrapperEnum e(c); !e.empty(); e.popFront()) {
if (e.front().key().kind != CrossCompartmentKey::StringWrapper)
AssertNotOnGrayList(&e.front().value().get().toObject());
}
}
#endif
FinishMarkingValidation(rt);
rt->gcLastGCTime = PRMJ_Now();
}
namespace {
/* ...while this class is to be used only for garbage collection. */
class AutoGCSession
{
JSRuntime* runtime;
AutoTraceSession session;
bool canceled;
public:
explicit AutoGCSession(JSRuntime* rt);
~AutoGCSession();
void cancel() { canceled = true; }
};
} /* anonymous namespace */
/* Start a new heap session. */
AutoTraceSession::AutoTraceSession(JSRuntime* rt, js::HeapState heapState)
: lock(rt),
runtime(rt),
prevState(rt->heapState)
{
JS_ASSERT(!rt->noGCOrAllocationCheck);
JS_ASSERT(!rt->isHeapBusy());
JS_ASSERT(heapState != Idle);
#ifdef JSGC_GENERATIONAL
JS_ASSERT_IF(heapState == MajorCollecting, rt->gcNursery.isEmpty());
#endif
// Threads with an exclusive context can hit refillFreeList while holding
// the exclusive access lock. To avoid deadlocking when we try to acquire
// this lock during GC and the other thread is waiting, make sure we hold
// the exclusive access lock during GC sessions.
JS_ASSERT(rt->currentThreadHasExclusiveAccess());
if (rt->exclusiveThreadsPresent()) {
// Lock the worker thread state when changing the heap state in the
// presence of exclusive threads, to avoid racing with refillFreeList.
#ifdef JS_THREADSAFE
AutoLockWorkerThreadState lock;
rt->heapState = heapState;
#else
MOZ_CRASH();
#endif
} else {
rt->heapState = heapState;
}
}
AutoTraceSession::~AutoTraceSession()
{
JS_ASSERT(runtime->isHeapBusy());
if (runtime->exclusiveThreadsPresent()) {
#ifdef JS_THREADSAFE
AutoLockWorkerThreadState lock;
runtime->heapState = prevState;
// Notify any worker threads waiting for the trace session to end.
WorkerThreadState().notifyAll(GlobalWorkerThreadState::PRODUCER);
#else
MOZ_CRASH();
#endif
} else {
runtime->heapState = prevState;
}
}
AutoGCSession::AutoGCSession(JSRuntime* rt)
: runtime(rt),
session(rt, MajorCollecting),
canceled(false)
{
runtime->gcIsNeeded = false;
runtime->gcInterFrameGC = true;
runtime->gcNumber++;
// It's ok if threads other than the main thread have suppressGC set, as
// they are operating on zones which will not be collected from here.
JS_ASSERT(!runtime->mainThread.suppressGC);
}
AutoGCSession::~AutoGCSession()
{
if (canceled)
return;
#ifndef JS_MORE_DETERMINISTIC
runtime->gcNextFullGCTime = PRMJ_Now() + GC_IDLE_FULL_SPAN;
#endif
runtime->gcChunkAllocationSinceLastGC = false;
#ifdef JS_GC_ZEAL
/* Keeping these around after a GC is dangerous. */
runtime->gcSelectedForMarking.clearAndFree();
#endif
/* Clear gcMallocBytes for all compartments */
for (ZonesIter zone(runtime, WithAtoms); !zone.done(); zone.next()) {
zone->resetGCMallocBytes();
zone->unscheduleGC();
}
runtime->resetGCMallocBytes();
}
AutoCopyFreeListToArenas::AutoCopyFreeListToArenas(JSRuntime* rt, ZoneSelector selector)
: runtime(rt),
selector(selector)
{
for (ZonesIter zone(rt, selector); !zone.done(); zone.next())
zone->allocator.arenas.copyFreeListsToArenas();
}
AutoCopyFreeListToArenas::~AutoCopyFreeListToArenas()
{
for (ZonesIter zone(runtime, selector); !zone.done(); zone.next())
zone->allocator.arenas.clearFreeListsInArenas();
}
class AutoCopyFreeListToArenasForGC
{
JSRuntime* runtime;
public:
AutoCopyFreeListToArenasForGC(JSRuntime* rt) : runtime(rt) {
JS_ASSERT(rt->currentThreadHasExclusiveAccess());
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next())
zone->allocator.arenas.copyFreeListsToArenas();
}
~AutoCopyFreeListToArenasForGC() {
for (ZonesIter zone(runtime, WithAtoms); !zone.done(); zone.next())
zone->allocator.arenas.clearFreeListsInArenas();
}
};
static void
IncrementalCollectSlice(JSRuntime* rt,
int64_t budget,
JS::gcreason::Reason gcReason,
JSGCInvocationKind gcKind);
static void
ResetIncrementalGC(JSRuntime* rt, const char* reason)
{
switch (rt->gcIncrementalState) {
case NO_INCREMENTAL:
return;
case MARK: {
/* Cancel any ongoing marking. */
AutoCopyFreeListToArenasForGC copy(rt);
rt->gcMarker.reset();
rt->gcMarker.stop();
for (GCCompartmentsIter c(rt); !c.done(); c.next()) {
ArrayBufferObject::resetArrayBufferList(c);
ResetGrayList(c);
}
for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
JS_ASSERT(zone->isGCMarking());
zone->setNeedsBarrier(false, Zone::UpdateIon);
zone->setGCState(Zone::NoGC);
}
rt->setNeedsBarrier(false);
AssertNeedsBarrierFlagsConsistent(rt);
rt->gcIncrementalState = NO_INCREMENTAL;
JS_ASSERT(!rt->gcStrictCompartmentChecking);
break;
}
case SWEEP:
rt->gcMarker.reset();
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next())
zone->scheduledForDestruction = false;
/* Finish sweeping the current zone group, then abort. */
rt->gcAbortSweepAfterCurrentGroup = true;
IncrementalCollectSlice(rt, SliceBudget::Unlimited, JS::gcreason::RESET, GC_NORMAL);
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_WAIT_BACKGROUND_THREAD);
rt->gcHelperThread.waitBackgroundSweepOrAllocEnd();
}
break;
default:
MOZ_ASSUME_UNREACHABLE("Invalid incremental GC state");
}
rt->gcStats.reset(reason);
#ifdef DEBUG
for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next())
JS_ASSERT(c->gcLiveArrayBuffers.empty());
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
JS_ASSERT(!zone->needsBarrier());
for (unsigned i = 0; i < FINALIZE_LIMIT; ++i)
JS_ASSERT(!zone->allocator.arenas.arenaListsToSweep[i]);
}
#endif
}
namespace {
class AutoGCSlice {
public:
AutoGCSlice(JSRuntime* rt);
~AutoGCSlice();
private:
JSRuntime* runtime;
};
} /* anonymous namespace */
AutoGCSlice::AutoGCSlice(JSRuntime* rt)
: runtime(rt)
{
/*
* During incremental GC, the compartment's active flag determines whether
* there are stack frames active for any of its scripts. Normally this flag
* is set at the beginning of the mark phase. During incremental GC, we also
* set it at the start of every phase.
*/
for (ActivationIterator iter(rt); !iter.done(); ++iter)
iter->compartment()->zone()->active = true;
for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
/*
* Clear needsBarrier early so we don't do any write barriers during
* GC. We don't need to update the Ion barriers (which is expensive)
* because Ion code doesn't run during GC. If need be, we'll update the
* Ion barriers in ~AutoGCSlice.
*/
if (zone->isGCMarking()) {
JS_ASSERT(zone->needsBarrier());
zone->setNeedsBarrier(false, Zone::DontUpdateIon);
} else {
JS_ASSERT(!zone->needsBarrier());
}
}
rt->setNeedsBarrier(false);
AssertNeedsBarrierFlagsConsistent(rt);
}
AutoGCSlice::~AutoGCSlice()
{
/* We can't use GCZonesIter if this is the end of the last slice. */
bool haveBarriers = false;
for (ZonesIter zone(runtime, WithAtoms); !zone.done(); zone.next()) {
if (zone->isGCMarking()) {
zone->setNeedsBarrier(true, Zone::UpdateIon);
zone->allocator.arenas.prepareForIncrementalGC(runtime);
haveBarriers = true;
} else {
zone->setNeedsBarrier(false, Zone::UpdateIon);
}
}
runtime->setNeedsBarrier(haveBarriers);
AssertNeedsBarrierFlagsConsistent(runtime);
}
static void
PushZealSelectedObjects(JSRuntime* rt)
{
#ifdef JS_GC_ZEAL
/* Push selected objects onto the mark stack and clear the list. */
for (JSObject** obj = rt->gcSelectedForMarking.begin();
obj != rt->gcSelectedForMarking.end(); obj++)
{
MarkObjectUnbarriered(&rt->gcMarker, obj, "selected obj");
}
#endif
}
static void
IncrementalCollectSlice(JSRuntime* rt,
int64_t budget,
JS::gcreason::Reason reason,
JSGCInvocationKind gckind)
{
JS_ASSERT(rt->currentThreadHasExclusiveAccess());
AutoCopyFreeListToArenasForGC copy(rt);
AutoGCSlice slice(rt);
bool destroyingRuntime = (reason == JS::gcreason::DESTROY_RUNTIME);
gc::State initialState = rt->gcIncrementalState;
int zeal = 0;
#ifdef JS_GC_ZEAL
if (reason == JS::gcreason::DEBUG_GC && budget != SliceBudget::Unlimited) {
/*
* Do the incremental collection type specified by zeal mode if the
* collection was triggered by RunDebugGC() and incremental GC has not
* been cancelled by ResetIncrementalGC.
*/
zeal = rt->gcZeal();
}
#endif
JS_ASSERT_IF(rt->gcIncrementalState != NO_INCREMENTAL, rt->gcIsIncremental);
rt->gcIsIncremental = budget != SliceBudget::Unlimited;
if (zeal == ZealIncrementalRootsThenFinish || zeal == ZealIncrementalMarkAllThenFinish) {
/*
* Yields between slices occurs at predetermined points in these modes;
* the budget is not used.
*/
budget = SliceBudget::Unlimited;
}
SliceBudget sliceBudget(budget);
if (rt->gcIncrementalState == NO_INCREMENTAL) {
rt->gcIncrementalState = MARK_ROOTS;
rt->gcLastMarkSlice = false;
}
if (rt->gcIncrementalState == MARK)
AutoGCRooter::traceAllWrappers(&rt->gcMarker);
switch (rt->gcIncrementalState) {
case MARK_ROOTS:
if (!BeginMarkPhase(rt)) {
rt->gcIncrementalState = NO_INCREMENTAL;
return;
}
if (!destroyingRuntime)
PushZealSelectedObjects(rt);
rt->gcIncrementalState = MARK;
if (rt->gcIsIncremental && zeal == ZealIncrementalRootsThenFinish)
break;
/* fall through */
case MARK: {
/* If we needed delayed marking for gray roots, then collect until done. */
if (!rt->gcMarker.hasBufferedGrayRoots()) {
sliceBudget.reset();
rt->gcIsIncremental = false;
}
bool finished = DrainMarkStack(rt, sliceBudget, gcstats::PHASE_MARK);
if (!finished)
break;
JS_ASSERT(rt->gcMarker.isDrained());
if (!rt->gcLastMarkSlice && rt->gcIsIncremental &&
((initialState == MARK && zeal != ZealIncrementalRootsThenFinish) ||
zeal == ZealIncrementalMarkAllThenFinish))
{
/*
* Yield with the aim of starting the sweep in the next
* slice. We will need to mark anything new on the stack
* when we resume, so we stay in MARK state.
*/
rt->gcLastMarkSlice = true;
break;
}
rt->gcIncrementalState = SWEEP;
/*
* This runs to completion, but we don't continue if the budget is
* now exhasted.
*/
BeginSweepPhase(rt, destroyingRuntime);
if (sliceBudget.isOverBudget())
break;
/*
* Always yield here when running in incremental multi-slice zeal
* mode, so RunDebugGC can reset the slice buget.
*/
if (rt->gcIsIncremental && zeal == ZealIncrementalMultipleSlices)
break;
/* fall through */
}
case SWEEP: {
bool finished = SweepPhase(rt, sliceBudget);
if (!finished)
break;
EndSweepPhase(rt, gckind, destroyingRuntime);
if (rt->gcSweepOnBackgroundThread)
rt->gcHelperThread.startBackgroundSweep(gckind == GC_SHRINK);
rt->gcIncrementalState = NO_INCREMENTAL;
break;
}
default:
JS_ASSERT(false);
}
}
IncrementalSafety
gc::IsIncrementalGCSafe(JSRuntime* rt)
{
JS_ASSERT(!rt->mainThread.suppressGC);
if (rt->keepAtoms())
return IncrementalSafety::Unsafe("keepAtoms set");
if (!rt->gcIncrementalEnabled)
return IncrementalSafety::Unsafe("incremental permanently disabled");
return IncrementalSafety::Safe();
}
static void
BudgetIncrementalGC(JSRuntime* rt, int64_t* budget)
{
IncrementalSafety safe = IsIncrementalGCSafe(rt);
if (!safe) {
ResetIncrementalGC(rt, safe.reason());
*budget = SliceBudget::Unlimited;
rt->gcStats.nonincremental(safe.reason());
return;
}
if (rt->gcMode() != JSGC_MODE_INCREMENTAL) {
ResetIncrementalGC(rt, "GC mode change");
*budget = SliceBudget::Unlimited;
rt->gcStats.nonincremental("GC mode");
return;
}
if (rt->isTooMuchMalloc()) {
*budget = SliceBudget::Unlimited;
rt->gcStats.nonincremental("malloc bytes trigger");
}
bool reset = false;
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
if (zone->gcBytes >= zone->gcTriggerBytes) {
*budget = SliceBudget::Unlimited;
rt->gcStats.nonincremental("allocation trigger");
}
if (rt->gcIncrementalState != NO_INCREMENTAL &&
zone->isGCScheduled() != zone->wasGCStarted())
{
reset = true;
}
if (zone->isTooMuchMalloc()) {
*budget = SliceBudget::Unlimited;
rt->gcStats.nonincremental("malloc bytes trigger");
}
}
if (reset)
ResetIncrementalGC(rt, "zone change");
}
/*
* Run one GC "cycle" (either a slice of incremental GC or an entire
* non-incremental GC. We disable inlining to ensure that the bottom of the
* stack with possible GC roots recorded in MarkRuntime excludes any pointers we
* use during the marking implementation.
*
* Returns true if we "reset" an existing incremental GC, which would force us
* to run another cycle.
*/
static MOZ_NEVER_INLINE bool
GCCycle(JSRuntime* rt, bool incremental, int64_t budget,
JSGCInvocationKind gckind, JS::gcreason::Reason reason)
{
AutoGCSession gcsession(rt);
/*
* As we about to purge caches and clear the mark bits we must wait for
* any background finalization to finish. We must also wait for the
* background allocation to finish so we can avoid taking the GC lock
* when manipulating the chunks during the GC.
*/
{
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_WAIT_BACKGROUND_THREAD);
rt->gcHelperThread.waitBackgroundSweepOrAllocEnd();
}
State prevState = rt->gcIncrementalState;
if (!incremental) {
/* If non-incremental GC was requested, reset incremental GC. */
ResetIncrementalGC(rt, "requested");
rt->gcStats.nonincremental("requested");
budget = SliceBudget::Unlimited;
} else {
BudgetIncrementalGC(rt, &budget);
}
/* The GC was reset, so we need a do-over. */
if (prevState != NO_INCREMENTAL && rt->gcIncrementalState == NO_INCREMENTAL) {
gcsession.cancel();
return true;
}
IncrementalCollectSlice(rt, budget, reason, gckind);
return false;
}
#ifdef JS_GC_ZEAL
static bool
IsDeterministicGCReason(JS::gcreason::Reason reason)
{
if (reason > JS::gcreason::DEBUG_GC &&
reason != JS::gcreason::CC_FORCED && reason != JS::gcreason::SHUTDOWN_CC)
{
return false;
}
if (reason == JS::gcreason::MAYBEGC)
return false;
return true;
}
#endif
static bool
ShouldCleanUpEverything(JSRuntime* rt, JS::gcreason::Reason reason, JSGCInvocationKind gckind)
{
// During shutdown, we must clean everything up, for the sake of leak
// detection. When a runtime has no contexts, or we're doing a GC before a
// shutdown CC, those are strong indications that we're shutting down.
return reason == JS::gcreason::DESTROY_RUNTIME ||
reason == JS::gcreason::SHUTDOWN_CC ||
gckind == GC_SHRINK;
}
namespace {
#ifdef JSGC_GENERATIONAL
class AutoDisableStoreBuffer
{
JSRuntime* runtime;
bool prior;
public:
AutoDisableStoreBuffer(JSRuntime* rt) : runtime(rt) {
prior = rt->gcStoreBuffer.isEnabled();
rt->gcStoreBuffer.disable();
}
~AutoDisableStoreBuffer() {
if (prior)
runtime->gcStoreBuffer.enable();
}
};
#else
struct AutoDisableStoreBuffer
{
AutoDisableStoreBuffer(JSRuntime*) {}
};
#endif
} /* anonymous namespace */
static void
Collect(JSRuntime* rt, bool incremental, int64_t budget,
JSGCInvocationKind gckind, JS::gcreason::Reason reason)
{
/* GC shouldn't be running in parallel execution mode */
JS_ASSERT(!InParallelSection());
JS_AbortIfWrongThread(rt);
/* If we attempt to invoke the GC while we are running in the GC, assert. */
JS_ASSERT(!rt->isHeapBusy());
if (rt->mainThread.suppressGC)
return;
TraceLogger* logger = TraceLoggerForMainThread(rt);
AutoTraceLog logGC(logger, TraceLogger::GC);
#ifdef JS_GC_ZEAL
if (rt->gcDeterministicOnly && !IsDeterministicGCReason(reason))
return;
#endif
JS_ASSERT_IF(!incremental || budget != SliceBudget::Unlimited, JSGC_INCREMENTAL);
AutoStopVerifyingBarriers av(rt, reason == JS::gcreason::SHUTDOWN_CC ||
reason == JS::gcreason::DESTROY_RUNTIME);
RecordNativeStackTopForGC(rt);
int zoneCount = 0;
int compartmentCount = 0;
int collectedCount = 0;
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
if (rt->gcMode() == JSGC_MODE_GLOBAL)
zone->scheduleGC();
/* This is a heuristic to avoid resets. */
if (rt->gcIncrementalState != NO_INCREMENTAL && zone->needsBarrier())
zone->scheduleGC();
zoneCount++;
if (zone->isGCScheduled())
collectedCount++;
}
for (CompartmentsIter c(rt, WithAtoms); !c.done(); c.next())
compartmentCount++;
rt->gcShouldCleanUpEverything = ShouldCleanUpEverything(rt, reason, gckind);
bool repeat = false;
do {
MinorGC(rt, reason);
/*
* Marking can trigger many incidental post barriers, some of them for
* objects which are not going to be live after the GC.
*/
AutoDisableStoreBuffer adsb(rt);
gcstats::AutoGCSlice agc(rt->gcStats, collectedCount, zoneCount, compartmentCount, reason);
/*
* Let the API user decide to defer a GC if it wants to (unless this
* is the last context). Invoke the callback regardless.
*/
if (rt->gcIncrementalState == NO_INCREMENTAL) {
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_GC_BEGIN);
if (JSGCCallback callback = rt->gcCallback)
callback(rt, JSGC_BEGIN, rt->gcCallbackData);
}
rt->gcPoke = false;
bool wasReset = GCCycle(rt, incremental, budget, gckind, reason);
if (rt->gcIncrementalState == NO_INCREMENTAL) {
gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_GC_END);
if (JSGCCallback callback = rt->gcCallback)
callback(rt, JSGC_END, rt->gcCallbackData);
}
/* Need to re-schedule all zones for GC. */
if (rt->gcPoke && rt->gcShouldCleanUpEverything)
JS::PrepareForFullGC(rt);
/*
* If we reset an existing GC, we need to start a new one. Also, we
* repeat GCs that happen during shutdown (the gcShouldCleanUpEverything
* case) until we can be sure that no additional garbage is created
* (which typically happens if roots are dropped during finalizers).
*/
repeat = (rt->gcPoke && rt->gcShouldCleanUpEverything) || wasReset;
} while (repeat);
if (rt->gcIncrementalState == NO_INCREMENTAL) {
#ifdef JS_THREADSAFE
EnqueuePendingParseTasksAfterGC(rt);
#endif
}
}
void
js::GC(JSRuntime* rt, JSGCInvocationKind gckind, JS::gcreason::Reason reason)
{
Collect(rt, false, SliceBudget::Unlimited, gckind, reason);
}
void
js::GCSlice(JSRuntime* rt, JSGCInvocationKind gckind, JS::gcreason::Reason reason, int64_t millis)
{
int64_t sliceBudget;
if (millis)
sliceBudget = SliceBudget::TimeBudget(millis);
else if (rt->gcHighFrequencyGC && rt->gcDynamicMarkSlice)
sliceBudget = rt->gcSliceBudget * IGC_MARK_SLICE_MULTIPLIER;
else
sliceBudget = rt->gcSliceBudget;
Collect(rt, true, sliceBudget, gckind, reason);
}
void
js::GCFinalSlice(JSRuntime* rt, JSGCInvocationKind gckind, JS::gcreason::Reason reason)
{
Collect(rt, true, SliceBudget::Unlimited, gckind, reason);
}
static bool
ZonesSelected(JSRuntime* rt)
{
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
if (zone->isGCScheduled())
return true;
}
return false;
}
void
js::GCDebugSlice(JSRuntime* rt, bool limit, int64_t objCount)
{
int64_t budget = limit ? SliceBudget::WorkBudget(objCount) : SliceBudget::Unlimited;
if (!ZonesSelected(rt)) {
if (JS::IsIncrementalGCInProgress(rt))
JS::PrepareForIncrementalGC(rt);
else
JS::PrepareForFullGC(rt);
}
Collect(rt, true, budget, GC_NORMAL, JS::gcreason::DEBUG_GC);
}
/* Schedule a full GC unless a zone will already be collected. */
void
js::PrepareForDebugGC(JSRuntime* rt)
{
if (!ZonesSelected(rt))
JS::PrepareForFullGC(rt);
}
JS_FRIEND_API(void)
JS::ShrinkGCBuffers(JSRuntime* rt)
{
AutoLockGC lock(rt);
JS_ASSERT(!rt->isHeapBusy());
if (!rt->useHelperThreads())
ExpireChunksAndArenas(rt, true);
else
rt->gcHelperThread.startBackgroundShrink();
}
void
js::MinorGC(JSRuntime* rt, JS::gcreason::Reason reason)
{
#ifdef JSGC_GENERATIONAL
TraceLogger* logger = TraceLoggerForMainThread(rt);
AutoTraceLog logMinorGC(logger, TraceLogger::MinorGC);
rt->gcNursery.collect(rt, reason, nullptr);
JS_ASSERT_IF(!rt->mainThread.suppressGC, rt->gcNursery.isEmpty());
#endif
}
void
js::MinorGC(JSContext* cx, JS::gcreason::Reason reason)
{
// Alternate to the runtime-taking form above which allows marking type
// objects as needing pretenuring.
#ifdef JSGC_GENERATIONAL
TraceLogger* logger = TraceLoggerForMainThread(cx->runtime());
AutoTraceLog logMinorGC(logger, TraceLogger::MinorGC);
Nursery::TypeObjectList pretenureTypes;
JSRuntime* rt = cx->runtime();
rt->gcNursery.collect(cx->runtime(), reason, &pretenureTypes);
for (size_t i = 0; i < pretenureTypes.length(); i++) {
if (pretenureTypes[i]->canPreTenure())
pretenureTypes[i]->setShouldPreTenure(cx);
}
JS_ASSERT_IF(!rt->mainThread.suppressGC, rt->gcNursery.isEmpty());
#endif
}
void
js::gc::GCIfNeeded(JSContext* cx)
{
JSRuntime* rt = cx->runtime();
#ifdef JSGC_GENERATIONAL
/*
* In case of store buffer overflow perform minor GC first so that the
* correct reason is seen in the logs.
*/
if (rt->gcStoreBuffer.isAboutToOverflow())
MinorGC(cx, JS::gcreason::FULL_STORE_BUFFER);
#endif
if (rt->gcIsNeeded)
GCSlice(rt, GC_NORMAL, rt->gcTriggerReason);
}
void
js::gc::FinishBackgroundFinalize(JSRuntime* rt)
{
rt->gcHelperThread.waitBackgroundSweepEnd();
}
AutoFinishGC::AutoFinishGC(JSRuntime* rt)
{
if (JS::IsIncrementalGCInProgress(rt)) {
JS::PrepareForIncrementalGC(rt);
JS::FinishIncrementalGC(rt, JS::gcreason::API);
}
gc::FinishBackgroundFinalize(rt);
}
AutoPrepareForTracing::AutoPrepareForTracing(JSRuntime* rt, ZoneSelector selector)
: finish(rt),
session(rt),
copy(rt, selector)
{
RecordNativeStackTopForGC(rt);
}
JSCompartment*
js::NewCompartment(JSContext* cx, Zone* zone, JSPrincipals* principals,
const JS::CompartmentOptions& options)
{
JSRuntime* rt = cx->runtime();
JS_AbortIfWrongThread(rt);
ScopedJSDeletePtr<Zone> zoneHolder;
if (!zone) {
zone = cx->new_<Zone>(rt);
if (!zone)
return nullptr;
zoneHolder.reset(zone);
zone->setGCLastBytes(8192, GC_NORMAL);
const JSPrincipals* trusted = rt->trustedPrincipals();
zone->isSystem = principals && principals == trusted;
}
ScopedJSDeletePtr<JSCompartment> compartment(cx->new_<JSCompartment>(zone, options));
if (!compartment || !compartment->init(cx))
return nullptr;
// Set up the principals.
JS_SetCompartmentPrincipals(compartment, principals);
AutoLockGC lock(rt);
if (!zone->compartments.append(compartment.get())) {
js_ReportOutOfMemory(cx);
return nullptr;
}
if (zoneHolder && !rt->zones.append(zone)) {
js_ReportOutOfMemory(cx);
return nullptr;
}
zoneHolder.forget();
return compartment.forget();
}
void
gc::MergeCompartments(JSCompartment* source, JSCompartment* target)
{
// The source compartment must be specifically flagged as mergable. This
// also implies that the compartment is not visible to the debugger.
JS_ASSERT(source->options_.mergeable());
JSRuntime* rt = source->runtimeFromMainThread();
AutoPrepareForTracing prepare(rt, SkipAtoms);
// Cleanup tables and other state in the source compartment that will be
// meaningless after merging into the target compartment.
source->clearTables();
// Fixup compartment pointers in source to refer to target.
for (CellIter iter(source->zone(), FINALIZE_SCRIPT); !iter.done(); iter.next()) {
JSScript* script = iter.get<JSScript>();
JS_ASSERT(script->compartment() == source);
script->compartment_ = target;
}
for (CellIter iter(source->zone(), FINALIZE_BASE_SHAPE); !iter.done(); iter.next()) {
BaseShape* base = iter.get<BaseShape>();
JS_ASSERT(base->compartment() == source);
base->compartment_ = target;
}
// Fixup zone pointers in source's zone to refer to target's zone.
for (size_t thingKind = 0; thingKind != FINALIZE_LIMIT; thingKind++) {
for (ArenaIter aiter(source->zone(), AllocKind(thingKind)); !aiter.done(); aiter.next()) {
ArenaHeader* aheader = aiter.get();
aheader->zone = target->zone();
}
}
// The source should be the only compartment in its zone.
for (CompartmentsInZoneIter c(source->zone()); !c.done(); c.next())
JS_ASSERT(c.get() == source);
// Merge the allocator in source's zone into target's zone.
target->zone()->allocator.arenas.adoptArenas(rt, &source->zone()->allocator.arenas);
target->zone()->gcBytes += source->zone()->gcBytes;
source->zone()->gcBytes = 0;
// Merge other info in source's zone into target's zone.
target->zone()->types.typeLifoAlloc.transferFrom(&source->zone()->types.typeLifoAlloc);
}
void
gc::RunDebugGC(JSContext* cx)
{
#ifdef JS_GC_ZEAL
JSRuntime* rt = cx->runtime();
int type = rt->gcZeal();
if (rt->mainThread.suppressGC)
return;
if (type == js::gc::ZealGenerationalGCValue)
return MinorGC(rt, JS::gcreason::DEBUG_GC);
PrepareForDebugGC(cx->runtime());
if (type == ZealIncrementalRootsThenFinish ||
type == ZealIncrementalMarkAllThenFinish ||
type == ZealIncrementalMultipleSlices)
{
js::gc::State initialState = rt->gcIncrementalState;
int64_t budget;
if (type == ZealIncrementalMultipleSlices) {
/*
* Start with a small slice limit and double it every slice. This
* ensure that we get multiple slices, and collection runs to
* completion.
*/
if (initialState == NO_INCREMENTAL)
rt->gcIncrementalLimit = rt->gcZealFrequency / 2;
else
rt->gcIncrementalLimit *= 2;
budget = SliceBudget::WorkBudget(rt->gcIncrementalLimit);
} else {
// This triggers incremental GC but is actually ignored by IncrementalMarkSlice.
budget = SliceBudget::WorkBudget(1);
}
Collect(rt, true, budget, GC_NORMAL, JS::gcreason::DEBUG_GC);
/*
* For multi-slice zeal, reset the slice size when we get to the sweep
* phase.
*/
if (type == ZealIncrementalMultipleSlices &&
initialState == MARK && rt->gcIncrementalState == SWEEP)
{
rt->gcIncrementalLimit = rt->gcZealFrequency / 2;
}
} else {
Collect(rt, false, SliceBudget::Unlimited, GC_NORMAL, JS::gcreason::DEBUG_GC);
}
#endif
}
void
gc::SetDeterministicGC(JSContext* cx, bool enabled)
{
#ifdef JS_GC_ZEAL
JSRuntime* rt = cx->runtime();
rt->gcDeterministicOnly = enabled;
#endif
}
void
gc::SetValidateGC(JSContext* cx, bool enabled)
{
JSRuntime* rt = cx->runtime();
rt->gcValidate = enabled;
}
void
gc::SetFullCompartmentChecks(JSContext* cx, bool enabled)
{
JSRuntime* rt = cx->runtime();
rt->gcFullCompartmentChecks = enabled;
}
#ifdef DEBUG
/* Should only be called manually under gdb */
void PreventGCDuringInteractiveDebug()
{
TlsPerThreadData.get()->suppressGC++;
}
#endif
void
js::ReleaseAllJITCode(FreeOp* fop)
{
#ifdef JS_ION
# ifdef JSGC_GENERATIONAL
/*
* Scripts can entrain nursery things, inserting references to the script
* into the store buffer. Clear the store buffer before discarding scripts.
*/
MinorGC(fop->runtime(), JS::gcreason::EVICT_NURSERY);
# endif
for (ZonesIter zone(fop->runtime(), SkipAtoms); !zone.done(); zone.next()) {
if (!zone->jitZone())
continue;
# ifdef DEBUG
/* Assert no baseline scripts are marked as active. */
for (CellIter i(zone, FINALIZE_SCRIPT); !i.done(); i.next()) {
JSScript* script = i.get<JSScript>();
JS_ASSERT_IF(script->hasBaselineScript(), !script->baselineScript()->active());
}
# endif
/* Mark baseline scripts on the stack as active. */
jit::MarkActiveBaselineScripts(zone);
jit::InvalidateAll(fop, zone);
for (CellIter i(zone, FINALIZE_SCRIPT); !i.done(); i.next()) {
JSScript* script = i.get<JSScript>();
jit::FinishInvalidation<SequentialExecution>(fop, script);
jit::FinishInvalidation<ParallelExecution>(fop, script);
/*
* Discard baseline script if it's not marked as active. Note that
* this also resets the active flag.
*/
jit::FinishDiscardBaselineScript(fop, script);
}
zone->jitZone()->optimizedStubSpace()->free();
}
#endif
}
/*
* There are three possible PCCount profiling states:
*
* 1. None: Neither scripts nor the runtime have count information.
* 2. Profile: Active scripts have count information, the runtime does not.
* 3. Query: Scripts do not have count information, the runtime does.
*
* When starting to profile scripts, counting begins immediately, with all JIT
* code discarded and recompiled with counts as necessary. Active interpreter
* frames will not begin profiling until they begin executing another script
* (via a call or return).
*
* The below API functions manage transitions to new states, according
* to the table below.
*
* Old State
* -------------------------
* Function None Profile Query
* --------
* StartPCCountProfiling Profile Profile Profile
* StopPCCountProfiling None Query Query
* PurgePCCounts None None None
*/
static void
ReleaseScriptCounts(FreeOp* fop)
{
JSRuntime* rt = fop->runtime();
JS_ASSERT(rt->scriptAndCountsVector);
ScriptAndCountsVector& vec = *rt->scriptAndCountsVector;
for (size_t i = 0; i < vec.length(); i++)
vec[i].scriptCounts.destroy(fop);
fop->delete_(rt->scriptAndCountsVector);
rt->scriptAndCountsVector = nullptr;
}
JS_FRIEND_API(void)
js::StartPCCountProfiling(JSContext* cx)
{
JSRuntime* rt = cx->runtime();
if (rt->profilingScripts)
return;
if (rt->scriptAndCountsVector)
ReleaseScriptCounts(rt->defaultFreeOp());
ReleaseAllJITCode(rt->defaultFreeOp());
rt->profilingScripts = true;
}
JS_FRIEND_API(void)
js::StopPCCountProfiling(JSContext* cx)
{
JSRuntime* rt = cx->runtime();
if (!rt->profilingScripts)
return;
JS_ASSERT(!rt->scriptAndCountsVector);
ReleaseAllJITCode(rt->defaultFreeOp());
ScriptAndCountsVector* vec = cx->new_<ScriptAndCountsVector>(SystemAllocPolicy());
if (!vec)
return;
for (ZonesIter zone(rt, SkipAtoms); !zone.done(); zone.next()) {
for (CellIter i(zone, FINALIZE_SCRIPT); !i.done(); i.next()) {
JSScript* script = i.get<JSScript>();
if (script->hasScriptCounts() && script->types) {
ScriptAndCounts sac;
sac.script = script;
sac.scriptCounts.set(script->releaseScriptCounts());
if (!vec->append(sac))
sac.scriptCounts.destroy(rt->defaultFreeOp());
}
}
}
rt->profilingScripts = false;
rt->scriptAndCountsVector = vec;
}
JS_FRIEND_API(void)
js::PurgePCCounts(JSContext* cx)
{
JSRuntime* rt = cx->runtime();
if (!rt->scriptAndCountsVector)
return;
JS_ASSERT(!rt->profilingScripts);
ReleaseScriptCounts(rt->defaultFreeOp());
}
void
js::PurgeJITCaches(Zone* zone)
{
#ifdef JS_ION
for (CellIterUnderGC i(zone, FINALIZE_SCRIPT); !i.done(); i.next()) {
JSScript* script = i.get<JSScript>();
/* Discard Ion caches. */
jit::PurgeCaches(script);
}
#endif
}
void
ArenaLists::normalizeBackgroundFinalizeState(AllocKind thingKind)
{
volatile uintptr_t* bfs = &backgroundFinalizeState[thingKind];
switch (*bfs) {
case BFS_DONE:
break;
case BFS_JUST_FINISHED:
// No allocations between end of last sweep and now.
// Transfering over arenas is a kind of allocation.
*bfs = BFS_DONE;
break;
default:
JS_ASSERT(!"Background finalization in progress, but it should not be.");
break;
}
}
void
ArenaLists::adoptArenas(JSRuntime* rt, ArenaLists* fromArenaLists)
{
// The other parallel threads have all completed now, and GC
// should be inactive, but still take the lock as a kind of read
// fence.
AutoLockGC lock(rt);
fromArenaLists->purge();
for (size_t thingKind = 0; thingKind != FINALIZE_LIMIT; thingKind++) {
#ifdef JS_THREADSAFE
// When we enter a parallel section, we join the background
// thread, and we do not run GC while in the parallel section,
// so no finalizer should be active!
normalizeBackgroundFinalizeState(AllocKind(thingKind));
fromArenaLists->normalizeBackgroundFinalizeState(AllocKind(thingKind));
#endif
ArenaList* fromList = &fromArenaLists->arenaLists[thingKind];
ArenaList* toList = &arenaLists[thingKind];
while (fromList->head != nullptr) {
// Remove entry from |fromList|
ArenaHeader* fromHeader = fromList->head;
fromList->head = fromHeader->next;
fromHeader->next = nullptr;
// During parallel execution, we sometimes keep empty arenas
// on the lists rather than sending them back to the chunk.
// Therefore, if fromHeader is empty, send it back to the
// chunk now. Otherwise, attach to |toList|.
if (fromHeader->isEmpty())
fromHeader->chunk()->releaseArena(fromHeader);
else
toList->insert(fromHeader);
}
fromList->cursor = &fromList->head;
}
}
bool
ArenaLists::containsArena(JSRuntime* rt, ArenaHeader* needle)
{
AutoLockGC lock(rt);
size_t allocKind = needle->getAllocKind();
for (ArenaHeader* aheader = arenaLists[allocKind].head;
aheader != nullptr;
aheader = aheader->next)
{
if (aheader == needle)
return true;
}
return false;
}
AutoMaybeTouchDeadZones::AutoMaybeTouchDeadZones(JSContext* cx)
: runtime(cx->runtime()),
markCount(runtime->gcObjectsMarkedInDeadZones),
inIncremental(JS::IsIncrementalGCInProgress(runtime)),
manipulatingDeadZones(runtime->gcManipulatingDeadZones)
{
runtime->gcManipulatingDeadZones = true;
}
AutoMaybeTouchDeadZones::AutoMaybeTouchDeadZones(JSObject* obj)
: runtime(obj->compartment()->runtimeFromMainThread()),
markCount(runtime->gcObjectsMarkedInDeadZones),
inIncremental(JS::IsIncrementalGCInProgress(runtime)),
manipulatingDeadZones(runtime->gcManipulatingDeadZones)
{
runtime->gcManipulatingDeadZones = true;
}
AutoMaybeTouchDeadZones::~AutoMaybeTouchDeadZones()
{
runtime->gcManipulatingDeadZones = manipulatingDeadZones;
if (inIncremental && runtime->gcObjectsMarkedInDeadZones != markCount) {
JS::PrepareForFullGC(runtime);
js::GC(runtime, GC_NORMAL, JS::gcreason::TRANSPLANT);
}
}
AutoSuppressGC::AutoSuppressGC(ExclusiveContext* cx)
: suppressGC_(cx->perThreadData->suppressGC)
{
suppressGC_++;
}
AutoSuppressGC::AutoSuppressGC(JSCompartment* comp)
: suppressGC_(comp->runtimeFromMainThread()->mainThread.suppressGC)
{
suppressGC_++;
}
AutoSuppressGC::AutoSuppressGC(JSRuntime* rt)
: suppressGC_(rt->mainThread.suppressGC)
{
suppressGC_++;
}
bool
js::UninlinedIsInsideNursery(JSRuntime* rt, const void* thing)
{
return IsInsideNursery(rt, thing);
}
#ifdef DEBUG
AutoDisableProxyCheck::AutoDisableProxyCheck(JSRuntime* rt
MOZ_GUARD_OBJECT_NOTIFIER_PARAM_IN_IMPL)
: count(rt->gcDisableStrictProxyCheckingCount)
{
MOZ_GUARD_OBJECT_NOTIFIER_INIT;
count++;
}
JS_FRIEND_API(void)
JS::AssertGCThingMustBeTenured(JSObject* obj)
{
JS_ASSERT((!IsNurseryAllocable(obj->tenuredGetAllocKind()) || obj->getClass()->finalize) &&
obj->isTenured());
}
JS_FRIEND_API(size_t)
JS::GetGCNumber()
{
JSRuntime* rt = js::TlsPerThreadData.get()->runtimeFromMainThread();
if (!rt)
return 0;
return rt->gcNumber;
}
JS::AutoAssertNoGC::AutoAssertNoGC()
: runtime(nullptr), gcNumber(0)
{
js::PerThreadData* data = js::TlsPerThreadData.get();
if (data) {
/*
* GC's from off-thread will always assert, so off-thread is implicitly
* AutoAssertNoGC. We still need to allow AutoAssertNoGC to be used in
* code that works from both threads, however. We also use this to
* annotate the off thread run loops.
*/
runtime = data->runtimeIfOnOwnerThread();
if (runtime)
gcNumber = runtime->gcNumber;
}
}
JS::AutoAssertNoGC::AutoAssertNoGC(JSRuntime* rt)
: runtime(rt), gcNumber(rt->gcNumber)
{
}
JS::AutoAssertNoGC::~AutoAssertNoGC()
{
if (runtime)
MOZ_ASSERT(gcNumber == runtime->gcNumber, "GC ran inside an AutoAssertNoGC scope.");
}
#endif
namespace js {
namespace gc {
AutoEnterIteration::AutoEnterIteration(JSRuntime *rt_) : rt(rt_)
{
++rt->numActiveZoneIters;
}
AutoEnterIteration::~AutoEnterIteration()
{
MOZ_ASSERT(rt->numActiveZoneIters);
--rt->numActiveZoneIters;
}
} /* namespace gc */
} /* namespace js */
