Version	Description	Authors
1.1.0-1	Doc Creation	Liu Chong, Fan Chenchen
1.3.0-1	Doc Upgrading, Features envent and fakeface are added.	Zang Junqing, Zhang Siming
1.4.0-1	Doc Upgradin, and the problem that the packaged .Arscene file cannot be uploaded has been modified	Zang Junqing
1.4.0-1	The document is updated, adding a combination with 2D foreground and 2D trigger.	Fan Chenchen

1、Overview

MeiShe SDK provides face sticker rendering engine, which helps users to add various face stikcers during capture. For users to design more personalized stickers, we have developed a helper plugin called AR Scene Editor. This document is to guide users to the correct use of the plugin and how to complete the design work.

2、Environment

Hardware：Windows PC / MAC

Software：Unity 5.6.3 or above，the following instruction is based on Unity 5.6.3

Notes

MeiShe SDK only provides the design plugin. For Unity, we suggest users purchase the legitimate software for use. Meishe Network Technology Co., Ltd. will not be responsible for any problems or losses arising from the use of pirated Unity software.

3、Description

AR Scene Editor can only produce the face sticker packages applicable to the SDK. This plugin provides a whole set of AR Scene file making scheme for the SDK, realizing the visualization of model, material, lighting, animation and physical system effects. The tool is currently in the process of being perfected, and the types of stickers that can be made include

A. 2D face sticker

B. 3D face sticker

Stickers with physical system support
3D trigger animation (new in 1.3.0).
Handwritten scripts realize the combination of foreground filters.

4、Model Creation Specification

Users can use MAYA,3Ds MAX, etc. as the 3D model design tool. We provide the standard face model (meishe_reference_head.fbx), All designs should be based on the standard face model, so that the designed stickers could accurately match the real human face and that the occlusion relation is right.

Notes

All specification referenced in this doc should be combination of letter, number and underline. Chinese characters are not allowed.

All path referenced in this doc should be combination of letter, number and underline. Chinese characters are not allowed.

1. Model Creation Specification

A.Users use the standard face model (meishe_reference_head.fbx) to create face sticker.

Notes:

1	Number of facets should be within 2W.
2	Model history should be removed and transform should be zero.
3	In Meishe SDK, the opposite side of normal direction could not be displayed, models should be sealed. If necessary, AR Scene supports double side display, but this function has not been integrated into the AR Scene plugin in version 1.1.0. And double side display could lower the efficiency of the SDK.

B.The models in the outline will be rendered from top to bottom, which needs to be structured according to the spatial context. Otherwise it will not display correctly when using transparent sticker, etc.

C. Use the common method to bind the model，functions like IK, restriction can`t be use on bone.

D. All models and bones can be put into one main group, if the main group does not exist, unity will automatically generate one which has the same name as the FBX.

Notes

Models and bones cannot have duplicate names, and the SDK cannot recognize duplicate names.

4.2、Sticker Creation Specification

A. stickers in JPG, PNG, JPG/PNG series（caf）are supported，transparent stickers are created in PNG or caf.

B. MeiShe SDK supports stickers in any aspect ratio, however ratio in 2ⁿ is suggested. And it should not exceed 2048x2048.

Notes

1	After sticker is applied to the model in 3D software, attribute Diffuse Color will not work at all, it won`t be read by the MeiShe SDK.
2	caf is the SDK`s internal format for image series, which can`t be read by other 3D software. After converting image series to caf, a companion image is generated automatically, which is used as the delegate of the caf in unity. When generating the sticker package, the delegate is then replaced by the real caf. For more information of caf creation, please refer to 8.2 2D Image Series Generation.

4.3、Export FBX

FBX of version 2013 is demanded.

Option embed media should be disabled, otherwise it might cause errors.

If the skeleton has its own animation, it needs to be baked frame by frame when exporting, the SDK will play the animation in a loop.

If the physical system needs to be used, no animation needs to be made after binding, it can be exported directly.

Notes

For the imported fbx file, pay attention to clear unnecessary history. The animation must be based on the Polygon model itself. The model should not be bound to an empty object for group animation. The SDK will not recognize similar locator nodes in Maya.

5、Installation and Project Creation

5.1、Installation of plugin

A. unity project creation.

Notes: Unity of version 5.6.3 and above is required and 5.6.3 is suggested. This doc is based on 5.6.3.

Open the newly created unity project, double click on AR Scene plugin (meishe_ARScene_editor-1.1.0.unitypackage), then click on Import to import the plugin.

Notes

File unitypackage need not be copied to unity project, double click on the AR Scene plugin is just OK.

5.2、AR Scene Generation

Open Meishe→AR Scene Editor in the menu bar.

Click on “Create a new AR Scene” to create an AR Scene scene.

Notes：Scenes created in other ways can`t generate AR Scene package.

Name the AR Scene and click on “Create” to create the folder.

Notes

AR Scene folder`s name is the name of the AR Scene package generated afterward. The 1.4.0 version of the Basic menu bar adds the Translation property, which can write the Chinese name of the prop.

The generated AR Scene is equipped with standard face model, light and camera.

Notes:

The camera is only used to observe the front 2D view. When the front 2D view needs to be observed, switch to the camera view for observation

The Layer Group is created to observe face movements and physical system effects in real time. Do not copy, change outline levels, or delete it. It is only used when the entire head section is required to be manipulated in a whole

Open the created AR Scene folder, copy the needed FBX and image stickers to this folder.

Notes

All types of stickers should undergo this operation. Although 2D stickers could be imported from other folders expect for the unity project folder, this is not suggested.

6、Unity 3D Sticker Creation Specification

6.1、Unity Model Creation Specification

A. Drag the FBX to the Hierarchy

Replace files in AR Scene folder to adjust or delete FBX and sticker.

If multiple FBX exist, their position order in the outline must be the same as their rendering order.

Notes:

1	Don`t change the order of the FBX`s sub layers. This should be completed in the 3D software. Though Unity will reorder sub layers when import, the rendering order set in the 3D software is not affected.
2	If the 3D model has certain occlusion relationship with the real human face in motion. A plain head model with opacity 0 should be created to test the occlusion. And the occluding layer should be placed above the occluded layer.

6.2、Unity Material Creation Specification

A. AR Scene plugin supports “no_shading” and “standard_lighting”. Users can switch the whole FBX`s material in “Shadings Mode” in FBX group.

Notes

Users can only switch the whole FBX`s material in “Shadings Mode” in FBX group. Material adjustment on sub layer of model cann`t be recognized by AR Scene.

B. In version 1.3.0 and after, Settings of Blending Mode an Animation are added, Animation setting is discussed in “6.4 Unity Animation Creation Specification”.

Shading Mode	No Shading
Shading Mode	Standard Shading
Blending Mode	Normal
	Multiply
	Subtract
	Screen
	Add
	Exclusion
Animation

C. Attributes of the two types of material are listed in the figure below.

Notes:

1	Tilling and Offset is invalid
2	Unity can`t recognize Reflection Map. Users need to select it manually.
3	In version 1.1.0, Opacity of “no shading” is invalid, if opacity is needed, the material should be set to “standard lighting”.
4	For image series sticker creation, please refer to “8.2 Image Series Sticker Generation”.

6.3、Unity Lighting Creation Specification

A. AR Scene has a light by default，users can create other lights manually.

B. AR Scene only recognize light attributes “Transform”, “Type”, “Color”, “Intensity”.

Notes

For now, MeiShe SDK only supports two lighting mode, “Directional” and “Point”.

6.4、Unity Animation Creation Specification

Select the original FBX in AR Scene folder, Set attribute “Animation Type” to “Legacy” under “Rig” in the “Inspector”. Set “Warp Mode” under “Animations” to “Loop”. Then click on “Apply”. In this way, Unity will play the animation effect previously created in the 3D software in loop.

Notes:

1	Animation is created in the 3D software. To create animation in Unity, please refer to “7.3 Preview”.
2	Do not alter the previously created FBX in Unity. Animations created in Unity can only be previewed, and can be adopted by MeiShe　SDK.
3	It happens that the preview in “Inspector” is not very accurate. The effect played in “game” is considered to be always accurate.

In version 1.3.0 and after, After selecting model in “Hierarchy”, users can set the “Animation” in the “Inspector” as following image.

If the imported FBX contains animation, the drop-down list of “Animation” will display the attached animation clips. It’s pretty straightforward for models with multiple animation clips.

In the image above, animation clip “a1” is chosen to be the active clip for the model. In “Show More Animation Fields”, users can set the loop modes and loop regions for all clips in the model.

Wrap Mode	Repeat: repeat
	Mirror: mirror repeat
	Constant: keep repeating the last frame
Repeat Pattern When wrap mode is “Repeat” or “Mirror”, this param takes effects, the default value is null, meaning lopping for the whole segment

When wrap mode is “Repeat” or “Mirror”, users can set the repeat region and repeat times.

For instance, the lifetime of a1 is 3000 milliseconds, the input is 1000;2000;2， among which “1000” is the start time to loop, “2000” is the end time of loop, “2” is the loop times. The model will render the animation in 1-1000 milliseconds of a1, then render the animation in 1000-2000 twice, and finally render animation in 2000-3000. If the repeat times is 0, it means after rendering the animation in 0-1000, the engine will render the animation in 1000-2000 until it stops.

7、Unity Physical System Creation Specification

7.1、Ensure the Hierarchy

Make sure all FBX layers are dragged to “Layer Group”

Notes:

1	Do not alter “Layer Group” and “meishe_reference_head”, Do not try to unlock the locked attributes.
2	If there is a need to scale or translate all the FBX or standard face model as a whole, users can create a new “GameObject” sub layer, and then put all relevant objects to it. Then scale or translation on “GameObject” will also affect objects belonging to it.

7.2、Load Bone Physics

Select the root bone, which should be the origin of all the motion units in the physics system. Click on “Add Bone Physics” to add “Bone Physics” script to bones.

Notes:

1	Load separate “Bone Physics” script to all units in the physics system.
2	The root motion bone is decided by the “Root”, default of which is the bone equipped with the script. “Root” can be changed to other bones in the same FBX, however the translation of it is calculated according to object equipped with the “Bone Physics” script. Therefor we suggest unifying the “Root” and the bone equipped with the “Bone Physics” script.

7.3、Preview

There is an embedded head animation in “Resources” folder. In order to preview, users need to drag to “Controller” named “Layer Group” to “Layer Group” and click “Play”.Select “Face Animation” to alter the animation in “Animation”,

Notes:

1	When playing in “Animation”, “Bone Physics” script does take effect.
2	Attributes changed during playback will not be saved after stopping the playback.
3	If there already exists an animation in the FBX, both the two animations will be played. Animation created in Unity can only be previewed, and AR Scene does not recognize it.

7.4、Plugin Tune

“Bone Physics” script has attributes “Damping”, “Elasticity”, “Stiffness”, “Inert”, “Gravity”, “Force”, etc.

“Damping”, “Elasticity”, “Stiffness”, “Inert” can be tuned by curve. The final value of these attribute is the curve value multiplying a basis value. Users need to set the basis value to 1 to realize the real curve effect.

Gravity only works when model is rotated away from the original status, while wind is always affecting the bone.

Notes

In current version of AR Scene and SDK, same gravity value does perform the same. It is suggested that gravity is added after all other effects to avoid mis-observation.

Freeze Axis has not been implemented。

8、Unity 2D Sticker Creation Specification

8.1、Static 2D Sticker Generation

A. Copy image to AR Scene folder.

B. Click on “Create Sticker With Image”, select the image in AR Scene folder to generate an image in the scene.

C. Tune the “Transform” of the image.

8.2、2D Image Series Sticker Generation

A. Select a sequence of frames, set resolution, frame rate, “Pose” to generate the caf file.

Output Image Format：image format, PNG, JPG is supported.

Output Image Width：image width

Output Image Height：image height

Frame Rate：frame rate，usually set to 15/1, namely 15 frames per second.

Default Loop Mode：animation loop mode

Poster Frame Index：the index of the PNG image which is chosen as the delegate of the series.

B. Copy the caf file to AR Scene folder to automatically generate the company PNG file of the caf, of which “Poster Frame Index” is already set. Click on “Create Sticker With Image” to preview the company PNG file. Tune the “Transform” as needed.

Notes

Users can also preview the image by straightly selecting the caf file. However, “Poster Frame Index” must be set, otherwise the first frame will be regarded as the poster. And if the first frame is transparent, no image will be displayed.

9、Event Trigger【1.3.0 new】

In version 1.3.0 and after, event trigger mechanism is added. Users an set trigger conditions to control model switching, short/long audio playback. 2d sticker can’t be controlled yet.

In window “Event”, users can click “Add Event” to add triggers, detailed tunable params are listed below the button. In order to set the trigger, users need to set the trigger condition (in the red block) and the trigger object (int eh blue block).

9.1、Trigger Condition

Trigger condition contains the following params.

Event Type	Action Increasing
	Action Decreasing
	Face Emerging
	Face Vanishing
	Timer
Action Type It takes effects only if the “Event Type” is “Action Increasing” or “Action Decreasing”.	Blink
	Lips Part
	Head Yaw
	Head Pitch
	Brow Jump
	Lip Corner Puller
Threshold: Amplitude of motion It takes effects only if the “Event Type” is “Action Increasing” or “Action Decreasing”. Its valid region is 0 – 1.
Min Interval: The minimum interval between two triggers.
Time: The time elapse triggered. It takes effects only if the “Event Type” is “Timer”.
Period: Period length in periodic switch. It takes effects only if the “Event Type” is “Timer”. 0 means infinite loop.
Count: Loop count It takes effects only if the “Event Type” is “Timer”. 0 means infinite loop.

Notes	The unit of Min Interval, Time and Period is milliseconds. 1 second equals to 1000 milliseconds.
	Time set by the “Timer Trigger” is counted on the basis of the time when the AR Scene package is loaded.

9.2、Trigger object switch

Type: Trigger type	Change Animation
	Play Sound
	Play Music
	Stop Music
Layer ID: Model name It takes effects only if the “Type” is “Change Animation”.
Animation Name: Animation name in the model It takes effects only if the “Type” is “Change Animation”.
Start Index: Sequence number in the cycle Default value is -1. Meaning this trigger doesn’t repeat.
Index Period The total number of triggers in the cycle
Timeout Single trigger delay trigger time length The starting point of Timeout is the moment when the "trigger condition" is reached most recently
Loops Cycles Only valid when Type is Play Sound and Play Music, 0 means infinite loop

Notes	When using the Timeout parameter, if the trigger action is made again within the specified time (the trigger condition is reached), the state of the timeout will be refreshed. If the trigger action is frequently made, the effect corresponding to the timeout may not be seen.
	The difference between Sound and Music. Sound is used to control short-duration sound effects, while Music controls background music. Music is generally the only one in the same effect package.

9.3、Action trigger example

An animation can have multiple trigger events. The following example uses Action Increasing and Face Emerging. Here we make a helmet wearing effect, recognize the face and wear the helmet, and trigger the lifting and falling of the helmet mask by opening the mouth. Make the helmet model in the 3D software, and make the required animation clips a1, a2, a3 in the same model.

a1 represents the cartoon clip with the helmet on the head;

a2 represents the animation clip of the helmet lifted up;

a3 represents the animation clip with the helmet visor dropped.

Import the helmet model and adjust the initial parameters of the model as shown below:

1：First of all, we need to automatically put on the helmet when the face appears, that is, apply the animation clip a1. Then add the first trigger Event1, set the Event Type to Face Emerging (face appears), and Add Trigger to add a trigger object. The trigger object settings are as follows:

2：Later, when you open your mouth for the first time, apply animation clip a2 to raise the mask; when you open your mouth for the second time, apply animation clip a3 to let the mask down. After that, opening your mouth continuously will make the animation loop between a2 and a3. Then add a second trigger Event2.

Add Trigger twice to set a2 and a3 as trigger objects respectively, and set the Index Period cycle period of both trigger objects to 2. Set the Start Index of a2 to 0, which means that a2 will be executed first in a single loop. Set the Start Index of a3 to 1, which means that a3 will be executed second in a single loop.

The settings are as follows:

9.4、Time trigger example

This example uses the helmet model in the previous example, selects the helmet props (regardless of whether there is a recognized face), and starts to perform the animation of the mask lifting and falling after 5 seconds.

1: Set the Time value to 5000, the helmet will start to perform the animation a2 when the helmet is lifted after 5 seconds,

2: Set the Period value to 2000, which means that the trigger interval of a single Trigger is 2 seconds. After 2 seconds, the helmet starts to execute the animation a3 of the mask falling, and then loops the animation effects of a2 and a3. As shown below:\

10、Fake Face function

Fake Face can fit a single image closely to the face, and enable the texture to change with changes in facial expressions. Usually used to make face painting and beauty makeup.

10.1、Fake Faceparameter

To make the Fake Face effect, first draw the texture with other software (for drawing, please refer to the model UV map fakeface_UV.png attached to the tool), and then use the plug-in to fit the texture to the face model. In addition to allowing the import of textures, the plug-in also provides adjustments to the lighting and overlay methods to achieve a more realistic effect. The specific parameters are as follows:

Shading Mode	No Shading
Shading Mode	Standard Shading
Fake Face Map
Blending Mode	Normal
	Multiply
	Subtract
	Screen
	Add
	Exclusion

10.2、Example

1. Check Fake Face in the Fake Face panel of the plugin。There will be a fakeface model without textures（White area of the screen）。

2. Click on the dot behind Fake Face Map, and add a map drawn according to the UV grid map

3. Select the appropriate overlay mode, the default is Normal11、AR Scene Generation

11.1、AR Scene Package Generation

A. Set values in “AR Scene Editor”,

AR Scene Package Version： Each update of the package without altering the ID will add one to the version.

Max Face： Max number of faces supported, 0 as default, no upper limits.

Use Z-Buffer：Enable Z buffer, which is enabled by default, If the rendering order doesn`t matter, users can disable it to enhance the efficiency of SDK.

Anti-Aliasing：Anti-aliasing。

Click on “Generate AR Scene Package”, select a path that is not is the Unity project, then AR Scene files, zipped AR Scene package, and unzipped AR Scene folder are generated.

Notes

“AR Scene Package Id” can be altered manually, but not all characters are supported, no space in the end.

12、Make a face sticker with 2D foreground

1. Use Meishe special effects tools to make a 9v16 ratio foreground filter, and select the album mode to export.

2. Decompress the resulting album compression package.

3. Copy the foreground material and "fx9v16.xml" in the album folder to the 3D model folder exported by unity, and then use the 3D model folder to operate, the album folder is no longer used

4. Open the "fx9v16.xml" file and delete the "uuid=" field

5. Open the "arscene9v16.xml" file and add the "<effect2D source="fx9v16.xml"/>" field

6. Compress the unity folder into a zip, and add the suffix ".1" to the compressed package name

注意	给压缩包而不是文件夹添加后缀，给文件夹添加后缀会导致生成的ARScene文件读取失败

Notes

Add a suffix to the compressed package instead of the folder. Adding a suffix to the folder will cause the generated ARScene file to fail to read

7、If you want to modify the effect of the ARScene, you need to open the "info.json" file, and each time you modify the value of version +1, the number of suffixes of the compressed package also needs to be correspondingly +1

8. If you need to change the ID, use "UUIDFactory.exe", select Without braces, click Generate, generate a random ID, click Copy to copy this ID, and replace the uuid in the info file with this ID

9. Put the compressed package into the "arscene" folder in the "SDK conversion tool PackageTools" folder, double-click the "arscene.bat" file or "arscene.bat-shortcut" to generate the arscene file.

13. Make the trigger effect of 2D foreground/2D face sticker

2D elements cannot be directly triggered, only 2D foreground or 2D face stickers can be written into the script to trigger the script.

13.1 2D foreground trigger

A. Copy the material and "fx9v16.xml" file in the album generated by the Meishe special effects tool to the folder exported by unity and modify the "fx9v16.xml" file(For specific operations, please refer to Steps 1, 2, 3, 4, and 5 of 12. Make a 2D foreground face sticker)

B. Add the "faceActionDetection="true"" field to the "arscene9v16.xml" file. This field indicates that the SDK will read the movement of the face. If this field is not added, the expression trigger will not be recognized. If the exported 3D model has a trigger function, this field will be included in the script.

C. Write the "disable="1"" field to the added 2D layer. This field indicates that this layer is hidden at the beginning, and will not appear until triggered

D. Write the "name=" field to the added 2D layer. This field is used to name the script that needs to be triggered in the trigger. The name can be any name you want. The source needs to correspond to the corresponding xml name. If the corresponding xml needs to be renamed, for example, when multiple scripts need to be triggered, the name needs to be renamed here.

E. Add the trigger event layer. For the writing and meaning of event specific parameters, please refer to the 9.Event trigger chapter and the attachment "arscene-desc-sample-event.xml".

Among them, "fxOn" means to trigger the script, and "fxOff" means to stop the script. The effect of the example in the figure is that the 2D foreground layer will be triggered when the mouth is opened to 0.3 degree, and it will stop after running for 2200ms.

F. Add the fields shown in the figure to the info, and the corresponding text prompt will appear when the face sticker is used. Pay attention to the field format, the comma cannot be omitted.

G. Use the folder to generate the arscene file. (For specific operations, refer to Steps 6, 7, 8, and 9 of 12. Make a 2D foreground face sticker)

13.2 Triggering of 2D face stickers

A. Use unity to generate 2D face stickers (for specific operations refer to 8. Unity 2D stickers production specifications)

C. Write the "disable="1"" field to the 3D layer. This field indicates that this layer is hidden at the beginning and will not appear until triggered. The name defaults to main3d, and you can modify its name when you need to trigger multiple 3D scripts. The source needs to correspond to the corresponding xml name. If the corresponding xml needs to be renamed, the name also needs to be renamed here.

。

D. Add the trigger event layer. For the writing and meaning of event specific parameters, please refer to the 9.Event trigger chapter and the attachment "arscene-desc-sample-event.xml".

Among them, "fxOn" means to trigger the script, and "fxOff" means to stop the script. The effect of the example in the figure is that the 2D face sticker will be triggered when the mouth is opened to 0.3 degree, and it will stop after running for 2200ms.

E. Add the fields shown in the figure to the info, and the corresponding text prompt will appear when the face sticker is used. Pay attention to the field format, the comma cannot be missing.

F. Use the folder to generate the arscene file. (For specific operations, refer to Steps 6, 7, 8, and 9 of 12. Make a 2D foreground face sticker)

14、Test in Mobile Device

Import the generated AR Scene package to the MeiShe SDK Demo. Test the effect and tune attributes in AR Scene plugin.

Scan the QR code to download the SDK Demo App.

Android Demo

IOS Demo

After installation, enter into the capture scene, download an arbitrary face sticker, and quit the app.

Link the phone to the computer，Import the generated AR Scene package to the following folder.

Android: NvStreamingSdk/Asset/ARScene/
IOS：Documents/Asset/ARScene/

Restart the app, enter into the capture scene, click on the face sticker to preview.

码农公寓

1、Overview

2、Environment

3、Description

4、Model Creation Specification

4.2、Sticker Creation Specification

4.3、Export FBX

5、Installation and Project Creation

5.1、Installation of plugin

5.2、AR Scene Generation

6、Unity 3D Sticker Creation Specification

6.1、Unity Model Creation Specification

6.2、Unity Material Creation Specification

6.3、Unity Lighting Creation Specification

6.4、Unity Animation Creation Specification

7、Unity Physical System Creation Specification

7.1、Ensure the Hierarchy

7.2、Load Bone Physics

7.3、Preview

7.4、Plugin Tune

8、Unity 2D Sticker Creation Specification

8.1、Static 2D Sticker Generation

8.2、2D Image Series Sticker Generation

9、Event Trigger【1.3.0 new】

9.1、Trigger Condition

9.2、Trigger object switch

9.3、Action trigger example

9.4、Time trigger example

10、Fake Face function

10.1、Fake Faceparameter

10.2、Example

11.1、AR Scene Package Generation

12、Make a face sticker with 2D foreground

13. Make the trigger effect of 2D foreground/2D face sticker

14、Test in Mobile Device

相关文章